Query language - ABBYY Documentation

A query specifies what words or combinations of words a document must contain. The query language used by extraction scripts is based on XML.

Basic query structure

A simple query looks like this:

<Request>
    <Query>
        <Contain>
            <Required>
                <Form><Text>first</Text></Form>
            </Required>
            <Optional>
                <Form><Text>second</Text></Form>
            </Optional>
            <Except>
                <Query>
                    <Contain>
                        <Required>
                            <Form><Text>third</Text></Form>
                        </Required>
                        <Optional>
                            <Form><Text>fourth</Text></Form>
                        </Optional>
                    </Contain>
                </Query>
            </Except>
        </Contain>
    </Query>
</Request>

The following elements make up a query:

Element	Description
`<Request>`	Root element of a query.
`<Query>`	Basic query. Contains a query tree that specifies the conditions the combination of words must meet. A basic query can be part of a combination of word forms in a higher-level query. Required.
`<Contain>`	Specifies what words or combinations of words a document must contain. A query can have only one `<Contain>` element. Required.

The <Contain> element can include any number of the following elements, in any order:

Element	Description
`<Required>`	Specifies a word form or subquery that a word combination must contain. Optional.
`<Optional>`	Specifies a word form or subquery that a word combination can contain. Optional.
`<Except>`	Specifies a word form or subquery that a word combination must not contain. Optional.

Each of these elements must contain one of the following:

Element	Description
`<Query>`	Specifies whether a string of word forms must be included in or excluded from the word combination.
`<Form>`	Specifies whether a word form must be included in or excluded from a word combination.

The <Contain> element must contain at least one <Required> or <Optional> element.

Attributes of the Contain element

Additional restrictions on a combination of word forms can be specified in the attributes of the <Contain> tag.

MinCount attribute

Specify the minimum number of items in a combination of word forms in a MinCount attribute. Each item is either a word form or a string of word forms returned by the subquery. The default value of the attribute is 1. The value of this parameter must not be greater than the total number of <Required> and <Optional> elements in the combination of word forms. This parameter is useful when a combination of word forms contains one or more <Optional> elements. The following query looks for either the first or the second parameter:

<Query>
    <Contain>
        <Optional>
            <Form><Text>first</Text></Form>
        </Optional>
        <Optional>
            <Form><Text>second</Text></Form>
        </Optional>
    </Contain>
</Query>

The following query finds at least three of the four words specified in the query, of which two words are required:

<Query>
    <Contain MinCount="3">
        <Optional>
            <Form><Text>US</Text></Form>
        </Optional>
        <Required>
            <Form><Text>President</Text></Form>
        </Required>
        <Optional>
            <Form><Text>Barack</Text></Form>
        </Optional>
        <Required>
            <Form><Text>Obama</Text></Form>
        </Required>
    </Contain>
</Query>

This query finds the following phrases: “US President Barack Obama,” “President Barack Obama,” and “US President Obama.”

KeepOrder attribute

To specify a fixed order of items in a combination of word forms, use the KeepOrder attribute of type Boolean. The default value of the attribute is false. The following query specifies that the order of items is fixed:

<Query>
    <Contain KeepOrder="true">
        <Required>
            <Form><Text>first</Text></Form>
        </Required>
        <Required>
            <Form><Text>second</Text></Form>
        </Required>
        <Required>
            <Form><Text>third</Text></Form>
        </Required>
    </Contain>
</Query>

The combination “first third second third” will not match the query, even though the query does contain a string of words arranged in the required order. KeepOrder also applies to any <Except> elements. Any words corresponding to the <Except> elements in a query must not occur between the words that they separate in the query. However, they can occur in any other positions beyond the fixed-order query strings. For example, if you modify the previous query by putting the word form “second” into an <Except> element instead of <Required>, a document containing the combination of word forms “first third second” will match the query (but the word form “second” will not be included in the result). If you then remove the KeepOrder attribute, a document containing the combination of word forms “first third second” will not be included in the result, because the word form “second” must not occur anywhere in the text of a document.

MinDistance and MaxDistance attributes

The MinDistance and MaxDistance attributes specify the minimum and maximum distances between words in a query. These attributes have no default values. If either of the two attributes is not specified, no distance limitations apply. Distance between words is measured in words and is calculated as the difference in the positions of the two corresponding words. The distance between two neighboring words equals 1, so the minimum value of either attribute is 1. MaxDistance must be greater than or equal to MinDistance. The distance between two strings of words is calculated as follows. If the strings do not overlap, the distance is calculated as the difference between the position of the left-most word in the right string and the position of the right-most word in the left string. If the strings overlap, the distance is assumed to be 0. For example, in the phrase “The quick brown fox jumped over the lazy dog,” the distance between the strings “quick fox” and “lazy dog” is 4, and the distance between the strings “quick fox” and “brown lazy dog” is 0. For <Except> elements, the distance between words is calculated as follows:

If KeepOrder="true", the word must not occur within the specified distance of the neighboring words in the string (that is, those words between which it occurs in the query). At the same time, the distance between the neighbors of the <Except> element must be within the specified range.
If KeepOrder="false", the word must not occur within the specified distance of any other word in the string.

The following query uses KeepOrder="true" and MaxDistance="2":

<Query>
    <Contain KeepOrder="true" MaxDistance="2">
        <Required>
            <Form><Text>sodium /Text></Form>
        </Required>
        <Except>
            <Form><Text>tetraborate</Text></Form>
        </Except>
        <Optional>
            <Form><Text>carborate</Text></Form>
        </Optional>
        <Optional>
            <Form><Text>sulfate</Text></Form>
        </Optional>
        <Required>
            <Form><Text>decahydrate</Text></Form>
        </Required>
    </Contain>
</Query>

This query finds phrases such as “sodium carborate decahydrate” and “sodium sulfate decahydrate.” The next query uses KeepOrder="false" and MaxDistance="2":

<Query>
    <Contain KeepOrder="false" MaxDistance="2">
        <Required>
            <Form><Text>sodium /Text></Form>
        </Required>
        <Except>
            <Form><Text>tetraborate</Text></Form>
        </Except>
        <Optional>
            <Form><Text>carborate</Text></Form>
        </Optional>
        <Optional>
            <Form><Text>sulfate</Text></Form>
        </Optional>
        <Required>
            <Form><Text>decahydrate</Text></Form>
        </Required>
    </Contain>
</Query>

This query finds only “sodium sulfate decahydrate,” because the word “tetraborate” is placed in the <Except> tags and the maximum distance between “tetraborate” and “sodium” is two words.

Form element

A query for one word form is specified using a <Form> element. This element can contain the following elements:

<Attributes> — Optional element that contains a query for the attributes of a word form. For more information, see When to use extraction scripts.
<Text> — Optional element that contains the Unicode text of a word form. If no text is specified, any word that matches the attributes query will match the query. The attributes query is required in this case.

Additional search conditions for a word form can be specified in the attributes of the <Form> tag.

SearchType attribute

The type of word form search can be specified in the SearchType attribute of the <Form> tag. This attribute can have the following values:

Value	Description
`AllFormsSearch`	Looks for all the forms of the specified word.
`ExactSearch`	Looks only for the specified form of the word.
`PrefixSearch`	Looks for any word forms prefixed by the specified string.
`FuzzySearch`	Performs a fuzzy search for the specified word. Useful if your texts may contain OCR errors and `ExactSearch` will not work. Can only be used for words containing at least 3 characters. For words 3 to 5 characters long, allows 1 OCR error; for words longer than 5 characters, allows up to 2 OCR errors.
`FuzzyPrefixSearch`	Performs a fuzzy search for any words prefixed by the specified string.

The SearchType attribute is optional and is set to AllFormsSearch by default.

CaseSensitive attribute

For case-sensitive searches, use the CaseSensitive attribute of the <Form> tag. This attribute is optional and is set to false by default. The following query uses the SearchType and CaseSensitive attributes of the <Form> tag:

<Query>
    <Contain KeepOrder="true">
        <Required>
            <Form SearchType="ExactSearch" CaseSensitive="true">
                <Text>WHO</Text>
            </Form>
        </Required>
    </Contain>
</Query>

This query looks for the acronym WHO in exactly this form, which helps avoid a large number of redundant results containing “who,” “whom,” or “whose.”

Attributes element

A query for word form attributes is a logical expression built with AND, OR, and NOT operators. NOT is a unary operator, whereas AND and OR are n-ary. The operands of this logical expression are values of type Bool. This logical expression is written in the form of a tree. The result of the query is the word forms in the text of a document that satisfy this logical expression. To get word form attributes, use an <Attributes> element. This element can contain the following elements:

Element	Description
`<Attribute>`	Contains the text of the required word attribute. It is a leaf on the logical expression tree.
`<Or>`	The OR operator. A node in the tree.
`<And>`	The AND operator. A node in the tree.
`<Not>`	The NOT operator. A node in the tree.

The <Not> element is constructed in the same way as the <Attributes> element and can contain only one of these elements. The <Or> and <And> elements must contain at least two of these elements. The <Attribute> tag has an optional SearchType attribute that specifies the type of attribute search. This attribute can have the following values:

ExactSearch — Looks for the attribute in exactly the form specified in the query.
PrefixSearch — Looks for any attributes prefixed by the specified text.

Search for attributes is always case-sensitive. The SearchType attribute is set to ExactSearch by default. Suppose you have a document where you have already identified:

NER objects, by calling the ExtractNerObjects function
Word forms from a user dictionary named dictionary, by calling the ExtractWordsFromUserDictionary function
All objects that satisfy a regular expression passed as a parameter, by calling the ExtractRegularExpression function

Assume the resulting collection of these objects is named regExp.

The name of the collection can be used in XML queries performed on the indexed document. The resulting collection itself can be accessed by its name.

A form attribute query then looks like this:

<Query>
    <Contain>
        <Required>
            <Form>
                <Attributes>
                    <Attribute>regExp1</Attribute>
                </Attributes>
            </Form>
        </Required>
        <Required>
            <Form>
                <Attributes>
                    <And>
                        <Attribute>NerOrg1</Attribute>
                        <Not>
                            <Attribute> dictionary1</Attribute>
                        </Not>
                    </And>
                </Attributes>
            </Form>
        </Required>
    </Contain>
</Query>

This query looks for a two-word phrase, where the first word must match the specified regular expression and the second is an allowed name of an organization. The digits after the attribute names single out the required words in detected multi-word objects that have been indexed with the respective attributes. For example, a regular expression named date may find a date in the format “May 31, 2019.” Then date1 corresponds to the word “May,” date2 corresponds to “31,” and date3 corresponds to “2019.”

FormSet element

A query for multiple word forms is specified using a <FormSet> element. This type of query combines multiple one-form queries using OR. It is equivalent to a <Query> query where all subqueries are optional word form queries. For a <FormSet> query, however, you can specify an attribute query common to all the forms. This makes for more efficient searches when ExactSearch is used to find all the word forms, there is an attribute query for each form, and all these attribute queries share some common fragment. A <FormSet> element contains the following elements:

<Attributes> — Optional element that contains a query for form attributes. This query is combined with form attribute queries using AND.
<Form> — Required element that contains a query for a word form. A <FormSet> element must contain at least one <Form> element.

Sample XML address extraction query

The result of an address extraction query is a text string containing the first word of the country, the first word of the street, the first word of the city, the first word of the state, and the first word of the ZIP code (in that order and provided they are no more than 5 words apart from one another). The consecutive words that make up a component (for example, a street name) are additionally numbered in the index, starting from 1.

var xmlQuery = "<Request> \
<Query> \
     <Contain MaxDistance=\"5\" KeepOrder=\"true\"> \
          <Optional> \
               <Form><Attributes><Attribute>NerCountry1</Attribute></Attributes></Form> \
          </Optional> \
          <Optional> \
               <Form><Attributes><Attribute>NerStreet1</Attribute></Attributes></Form> \
          </Optional> \
          <Optional> \
               <Form><Attributes><Attribute>NerCity1</Attribute></Attributes></Form> \
          </Optional> \
          <Optional> \
               <Form><Attributes><Attribute>NerState1</Attribute></Attributes></Form> \
          </Optional> \
          <Required> \
               <Form><Attributes><Attribute>NerZipCode1</Attribute></Attributes></Form> \
          </Required> \
     </Contain> \
</Query> \
</Request>";

The result of an address extraction query is recorded into a repeating xmlQueryResult field:

this.RunQueryAndSaveToField( xmlQuery, "query", "xmlQueryResult" );

​Basic query structure

​Attributes of the Contain element

​MinCount attribute

​KeepOrder attribute

​MinDistance and MaxDistance attributes

​Form element

​SearchType attribute

​CaseSensitive attribute

​Attributes element

​FormSet element

​Sample XML address extraction query

Basic query structure

Attributes of the Contain element

MinCount attribute

KeepOrder attribute

MinDistance and MaxDistance attributes

Form element

SearchType attribute

CaseSensitive attribute

Attributes element

FormSet element

Sample XML address extraction query