diff --git a/.gitignore b/.gitignore index 97d7d8bbe7..14d2971b27 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,10 @@ temp tmp *.pdf *.swp +<<<<<<< HEAD +# ignore WebStorm IDE settings files +.idea/ +======= +# Ignore temporary files from IntelliJ editors +.idea/ +>>>>>>> vulcan diff --git a/content/admin/picts/TsAndCs01.png b/content/admin/picts/TsAndCs01.png index 86842144c2..ef9937d008 100644 Binary files a/content/admin/picts/TsAndCs01.png and b/content/admin/picts/TsAndCs01.png differ diff --git a/content/admin/picts/add-server-new2.png b/content/admin/picts/add-server-new2.png index 6e21428dca..a15262813b 100644 Binary files a/content/admin/picts/add-server-new2.png and b/content/admin/picts/add-server-new2.png differ diff --git a/content/admin/picts/auto-compact-defaultNewUI.png b/content/admin/picts/auto-compact-defaultNewUI.png index df18c51c09..cf42cc8473 100644 Binary files a/content/admin/picts/auto-compact-defaultNewUI.png and b/content/admin/picts/auto-compact-defaultNewUI.png differ diff --git a/content/admin/picts/configureNewCluster01.png b/content/admin/picts/configureNewCluster01.png index 2d23797dcb..59b38c919f 100644 Binary files a/content/admin/picts/configureNewCluster01.png and b/content/admin/picts/configureNewCluster01.png differ diff --git a/content/admin/picts/dashboard01.png b/content/admin/picts/dashboard01.png index 11a7ce344c..9657d7ea14 100644 Binary files a/content/admin/picts/dashboard01.png and b/content/admin/picts/dashboard01.png differ diff --git a/content/admin/picts/ephemeralBucketsReprovisioningInterface.png b/content/admin/picts/ephemeralBucketsReprovisioningInterface.png new file mode 100644 index 0000000000..0ffd0d75de Binary files /dev/null and b/content/admin/picts/ephemeralBucketsReprovisioningInterface.png differ diff --git a/content/admin/picts/graceful-failover.png b/content/admin/picts/graceful-failover.png index f5fd18d8e1..07e5d22b66 100644 Binary files a/content/admin/picts/graceful-failover.png and b/content/admin/picts/graceful-failover.png differ diff --git a/content/admin/picts/joinClusterMemoryQuotaSaved.png b/content/admin/picts/joinClusterMemoryQuotaSaved.png index 6d58162912..e6973dee5e 100644 Binary files a/content/admin/picts/joinClusterMemoryQuotaSaved.png and b/content/admin/picts/joinClusterMemoryQuotaSaved.png differ diff --git a/content/admin/picts/joinClusterServiceCheckboxes.png b/content/admin/picts/joinClusterServiceCheckboxes.png index 3e6f8ba0a9..511912c7a2 100644 Binary files a/content/admin/picts/joinClusterServiceCheckboxes.png and b/content/admin/picts/joinClusterServiceCheckboxes.png differ diff --git a/content/admin/picts/joinExistingNewServiceSettings.png b/content/admin/picts/joinExistingNewServiceSettings.png index e1cad18128..c182e3769c 100644 Binary files a/content/admin/picts/joinExistingNewServiceSettings.png and b/content/admin/picts/joinExistingNewServiceSettings.png differ diff --git a/content/admin/picts/joinWithCustomConfig.png b/content/admin/picts/joinWithCustomConfig.png index 303ab504af..22a8f9dd94 100644 Binary files a/content/admin/picts/joinWithCustomConfig.png and b/content/admin/picts/joinWithCustomConfig.png differ diff --git a/content/admin/picts/joinWithDefaultConfig.png b/content/admin/picts/joinWithDefaultConfig.png new file mode 100644 index 0000000000..d5c7de0dd7 Binary files /dev/null and b/content/admin/picts/joinWithDefaultConfig.png differ diff --git a/content/admin/picts/registerForUpdates01.png b/content/admin/picts/registerForUpdates01.png index 3931b6d12d..6034985402 100644 Binary files a/content/admin/picts/registerForUpdates01.png and b/content/admin/picts/registerForUpdates01.png differ diff --git a/content/admin/picts/reset-quota.png b/content/admin/picts/reset-quota.png index 18e9b1f7a9..bbac7db110 100644 Binary files a/content/admin/picts/reset-quota.png and b/content/admin/picts/reset-quota.png differ diff --git a/content/admin/picts/setUpNewCluster01.png b/content/admin/picts/setUpNewCluster01.png index 2589d75aad..ce84fb4cb1 100644 Binary files a/content/admin/picts/setUpNewCluster01.png and b/content/admin/picts/setUpNewCluster01.png differ diff --git a/content/admin/picts/settings-autofailover.png b/content/admin/picts/settings-autofailover.png index 8ee2047ee4..6e3e4efade 100644 Binary files a/content/admin/picts/settings-autofailover.png and b/content/admin/picts/settings-autofailover.png differ diff --git a/content/admin/picts/ui-cluster.png b/content/admin/picts/ui-cluster.png index ca0a09751e..2e22580920 100644 Binary files a/content/admin/picts/ui-cluster.png and b/content/admin/picts/ui-cluster.png differ diff --git a/content/admin/picts/ui-services.png b/content/admin/picts/ui-services.png index 540d350c27..b5ec1d4c0e 100644 Binary files a/content/admin/picts/ui-services.png and b/content/admin/picts/ui-services.png differ diff --git a/content/admin/picts/vbucket-resources.png b/content/admin/picts/vbucket-resources.png index 1f3b6bed6a..395742dba3 100644 Binary files a/content/admin/picts/vbucket-resources.png and b/content/admin/picts/vbucket-resources.png differ diff --git a/content/admin/picts/web-console.png b/content/admin/picts/web-console.png index 993f5c595e..82089c9930 100644 Binary files a/content/admin/picts/web-console.png and b/content/admin/picts/web-console.png differ diff --git a/content/admin/picts/welcome.png b/content/admin/picts/welcome.png index f4f82beac4..bdc2e9d137 100644 Binary files a/content/admin/picts/welcome.png and b/content/admin/picts/welcome.png differ diff --git a/content/admin/ui-intro.dita b/content/admin/ui-intro.dita index 9e6c268e04..d90498a4fe 100644 --- a/content/admin/ui-intro.dita +++ b/content/admin/ui-intro.dita @@ -8,7 +8,7 @@ environment. -

The redesigned 5.0 interface for Couchbase Server’s Web Console offers a new, modern look on +

The interface for Couchbase Server’s Web Console offers a new, modern look on usability in a browser-based application. Re-designed for intelligence, comfort, and speed, you will see a clean new look and experience a streamlined interface to Couchbase’s administration and development platform that optimizes your common tasks and workflows.

@@ -43,19 +43,27 @@ Data Service - The number of data nodes that have data service running. + The number of data nodes that have Data Service running. - GSI Service - The number of data nodes that have Indexer service running. + Index Service + The number of data nodes that have the Indexe Service running. FTS Service - The number of data nodes that have full text search (FTS) service running. + The number of data nodes that have the Full Text Search (FTS) Service running. Query Service - The number of data nodes that have query service running. + The number of data nodes that have the Query Service running. + + + Eventing Service + The number of data nodes that have the Eventing Service running. + + + Analytics Service + The number of data nodes that have the Analytics Service running. XDCR diff --git a/content/administrator-guide.ditamap b/content/administrator-guide.ditamap index 917d20064f..1cde086c44 100644 --- a/content/administrator-guide.ditamap +++ b/content/administrator-guide.ditamap @@ -16,6 +16,7 @@ + @@ -26,6 +27,7 @@ + @@ -39,6 +41,7 @@ + diff --git a/content/analytics/1_intro.html b/content/analytics/1_intro.html index d47f58dc72..582c6866ba 100644 --- a/content/analytics/1_intro.html +++ b/content/analytics/1_intro.html @@ -3,18 +3,17 @@ - Introduction
-

What’s SQL++?

+

This document is intended as a reference guide to the full syntax and semantics of the SQL++ Query Language, a SQL-inspired language for working with semistructured data. SQL++ has much in common with SQL, but there are also differences due to the data model that the language is designed to serve. (SQL -was designed in the 1970’s for interacting with the flat, schema-ified world of +was designed in the 1970's for interacting with the flat, schema-ified world of relational databases, while SQL++ is designed for the nested, schema-less/schema-optional world of modern NoSQL systems.) In particular, SQL++ in the context of Couchbase Analytics is intended for working with the JSON data model.

diff --git a/content/analytics/2_expr.html b/content/analytics/2_expr.html index 58614c9675..cfd8a53f0f 100644 --- a/content/analytics/2_expr.html +++ b/content/analytics/2_expr.html @@ -3,12 +3,11 @@ - Expressions
-

Expressions

+

This section lists all supported DDL statements in Couchbase Analytics.

@@ -16,6 +15,11 @@

Data Definition Language (DDL)

to execute DDL statements (Admin, Cluster Admin, or Analytics Manager).
+
Note: No `CREATE`/`DROP` or `CONNECT`/`DISCONNECT` statement can +be executed while the cluster topology changes (e.g. during rebalance). The evaluation of such DDL statements will fail +and it can be reattempted after the topology change has completed. +
+

Statements

Statement       ::= ( ( SingleStatement )? ( ";" )+ )* <EOF>
@@ -28,7 +32,7 @@ 
Statements

In addition to queries, the Couchbase Analytics implementation of SQL++ supports statements for data definition and to -connect Couchbase Analytics to Couchbase buckets.

+connect Couchbase Analytics to Data Service buckets.

Creating and Dropping

@@ -41,7 +45,7 @@

Creating and Droppi

-

The CREATE statement is used to create (Analytics) buckets, shadow datasets, and indexes.

+

The CREATE statement is used to create (Analytics) buckets, datasets, and indexes.

Buckets

@@ -57,57 +61,62 @@

Buckets

Note: For Bucket DDL statments, in addition to having one of the -roles required to execute DDL statements, a user must also have bucket data read permission on the data service bucket. +roles required to execute DDL statements, a user must also have bucket data read permission on the Data Service bucket.
-

A (Analytics) bucket is a proxy for a bucket on a cluster of Data service nodes. +

A (Analytics) bucket is a proxy for a bucket on a cluster of Data Service nodes. It represents this bucket in all other DDL operations. The CREATE BUCKET statement creates such a (Analytics) bucket. Parameters for the bucket can be provided though a parameter object. Currently 2 parameters are supported:

    -
  • name a string for the name of the bucket used on the Data service nodes and
    • -
  • nodes an array or IP addresses or node names for the Data service nodes. However, you should not pass -this parameter to the statement if the bucket resides in the same Couchbase instance.
    • +
  • name a string for the name of the bucket used on the Data Service nodes and
    • +
  • nodes an array or IP addresses or node names for the Data Service nodes. However, you should not pass + this parameter to the statement if the bucket resides in the same Couchbase instance.

Both parameters are optional - the default name is the name of the bucket and the default value for nodes -are the Data service nodes of the same Couchbase instance.

+are the Data Service nodes of the same Couchbase instance.

-
Example
+
Examples
+ +
CREATE BUCKET `beer-sample`;
 
-
CREATE BUCKET beerbucket WITH {
+CREATE BUCKET beerbucket WITH {
   "name": "beer-sample",
-  "nodes": ["localhost"]
+  "nodes": "192.168.1.1"
 };
 

 

 
-
This example creates a new (Analytics) bucket beerbucket that represents the bucket beer-sample on the Data service node running on localhost.

+
The first example creates a new (Analytics) bucket that represents the bucket beer-sample on the Data Service of the
+same Couchbase instance.
+The second example creates a new bucket beerbucket that represents the bucket beer-sample on the Data Service of a
+Couchbase instance that can be found on 192.168.1.1.

 
-
Shadow Datasets

+
Datasets

 
-
DatasetSpecification ::= "SHADOW" "DATASET" ( IfNotExists )? ( QualifiedName )?
+
DatasetSpecification ::= "DATASET" ( IfNotExists )? ( QualifiedName )?
                          "ON" Identifier ( "WHERE" Expression )?
 

 

 
-
Shadow datasets contain a shadow of the data of a bucket.
+
Datasets contain a shadow of the data of a bucket.
 They are connected to a bucket and are updated as the bucket gets updated.
-A Shadow dataset can contain the full content of the dataset or a filtered subset.
-Multiple shadow datasets can shadow the same bucket.
-A shadow dataset is created ON a previously created BUCKET with an optional filter expression.
+A dataset can contain the full content of the bucket or a filtered subset.
+Multiple datasets can shadow the same bucket.
+A dataset is created ON a previously created BUCKET with an optional filter expression.
 The name of the dataset is optional and defaults to the name of the bucket.

 
-
Examples

+
Examples

 
-
CREATE SHADOW DATASET beers ON beerbucket WHERE type = "beer";
-CREATE SHADOW DATASET breweries ON beerbucket WHERE type = "brewery";
+
CREATE DATASET beers ON `beer-sample` WHERE type = "beer";
+CREATE DATASET breweries ON `beer-sample` WHERE type = "brewery";
 

 

 
-
This example creates 2 shadow datasets beers and breweries on the previously created beerbucket and filters
+
This example creates 2 datasets beers and breweries on the previously created beer-sample bucket and filters
 the content for each dataset by the value of the type field of the record.

 
 
Indexes

@@ -122,14 +131,14 @@ 
Indexes

 The QualifiedName in the IndexSpecification identifies the dataset on which the index is built.
 The FieldSpecifications consist of a QualifiedName that specifies a field path into the
 indexed JSON document and a type identifier.
-To create an index on a shadow dataset,
-the bucket corresponding to the shadow dataset must be in the disconnected state.
-There may be more than one secondary index created on the same shadow dataset.
+To create an index on a dataset,
+the bucket corresponding to the dataset must be in the disconnected state.
+There may be more than one secondary index created on the same dataset.
 Each index name must be unique within a dataset.
 Creating index fails if there is an existing index with the same name
 in the target dataset and ‘IF NOT EXISTS’ is not specified.

 
-
For each JSON document ingested into a shadow dataset, the system computes the indexed key for each index.
+
For each JSON document ingested into a dataset, the system computes the indexed key for each index.
 The index key of a secondary index is computed as follows:

 
 
 
-
For case 3, password is required if the bucket requires a password while username should not be provided.
-The default password is the empty string.

-
-
Example

+
Example

 
-
CONNECT BUCKET beerbucket WITH { "timeout": 2500 };
+
CONNECT BUCKET `beer-sample` WITH { "timeout": 2500 };
 

 

 
-
This example connects all shadow datasets that were previously created on bucket beerbucket to the Data service
-nodes using !@# as the password and a timeout of 2500 ms.

+
This example connects all datasets that were previously created on bucket beer-sample to the Data Service
+nodes using a timeout of 2500 ms.

 
 
DisconnectStatement ::= "DISCONNECT" "BUCKET" QualifiedName  ( "IF" "CONNECTED" )?
 

@@ -254,13 +262,14 @@ 
Example

 
The DISCONNECT statement is the inverse of the CONNECT statement.
 It stops shadowing all datasets that were created on a BUCKET and disconnects the bucket.

 
-
Example

+
Example

 
-
DISCONNECT BUCKET beerbucket IF CONNECTED;
+
DISCONNECT BUCKET `beer-sample` IF CONNECTED;
 

 

 
-
This example stops shadowing all datasets that were created on the bucket beerbucket and disconnects the bucket if the bucket is connected.

+
This example stops shadowing all datasets that were created on the bucket beer-sample and disconnects the bucket if
+the bucket is connected.

 
 
 
 
SQL++ offers the following key advancements beyond N1QL:

-
 
    
-  
  • JOIN:  SQL++ supports the ANSI join syntax and allows joins on any condition expressions over datasets, arrays,
-      or subqueries;
    • 
-  
  • GROUP BY: In SQL++, in addition to a set of aggregate functions as in standard SQL, the groups created by the
-GROUP BY clause are directly usable in nested queries and/or to obtain nested results;
    • 
-  
  • Subquery: Any valid SQL++ query can be a valid subquery.
    • 
+  
  • WITH:  SQL++ supports the ANSI SQL WITH clause to allow the definition of inlined views or variables of primitive types to simplify complex query construction;
    • 
+  
  • JOIN:  SQL++ supports the ANSI join syntax and allows joins on any condition expressions over datasets, arrays, or subqueries;
    • 
+  
  • GROUP BY: In SQL++, in addition to a set of aggregate functions as in standard SQL, the groups created by the GROUP BY clause are directly usable in nested queries and/or to obtain nested results;
    • 
+  
  • Subquery: Any valid SQL++ query can be used as a subquery.
    • 
 

 
 
For N1QL users, the following matrix is a quick N1QL compatibility cheat sheet for SQL++.

@@ -56,11 +53,6 @@ 
SQL++ vs. N1QL

       SELECT * FROM user u 
LEFT OUTER NEST orders orders 
ON KEYS ARRAY s.order_id FOR s IN u.order_history END;
       SELECT u, (SELECT VALUE o FROM u.order_history s, orders o WHERE meta(o).id = s.order_id) orders 
FROM users u;
     
-    
-      IS VALUED
-      foo is VALUED
-      foo IS NOT UNKNOWN
-    
     
       ARRAY
       ARRAY i FOR i IN [1, 2] END
@@ -72,25 +64,47 @@ 
SQL++ vs. N1QL

       arr[0]
     
     
-      INSERT/UPSERT/DELETE
-      Supported
-      Unsupported
+      LIMIT l OFFSET o
+      Allows OFFSET without LIMIT
+      Doesn't support OFFSET without LIMIT
+    
+    
+      UNION, INTERSECT, and EXCEPT
+      All three are supported (with ALL and DISTINCT variants)
+      Only UNION ALL is supported (and necessary for query expressability)
+    
+    
+      OUTER JOIN
+      Both LEFT and RIGHT OUTER JOIN supported
+      Only LEFT OUTER JOIN supported (and necessary for query expressability)
+    
+    
+      <, <=, =, etc. operators
+      Can compare either complex values or scalar values
+      Only scalar values may be compared
     
     
       CREATE INDEX
       Supported
-      Unsupported
+      Supported but different (e.g., typed)
+    
+    
+      INSERT/UPSERT/DELETE
+      Supported
+      Unsupported (by design)
     
   
 
 
-
SQL++ generalizes N1QL’s syntax constructs such as USE KEYS, ON KEYS, ON KEY, NEST,
-LEFT OUTER NEST and ARRAY and hence relaxes the index or primary key
-restrictions for acceptable queries.
-Note that INSER/UPSERT/DELETE are not supported in the Couchbase Analytics Service.
-If you want to do any data mutations, do that through the Couchbase Server SDK
-or N1QL, and the mutations will be instantly, automatically synchronized into the Couchbase Analytics
-Service.

+
SQL++ generalizes N1QL's syntax constructs such as USE KEYS, ON KEYS, ON KEY, NEST,
+LEFT OUTER NEST and ARRAY and thus eliminates cases where must-be-indexed or must-use-keys
+restrictions are required for certain N1QL queries or expressions to be acceptable.
+In addition, the general composability of SQL++ queries eliminates the need for some of N1QL's
+special syntax; for example, SQL++ does not require or support the IN/WITHIN subclauses of
+N1QL's existential (SOME, ANY, or EVERY) expressions.
+Note that INSERT/UPSERT/DELETE are not supported at all in the Couchbase Analytics Service.
+Data is mutated in Couchbase Server, using the Couchbase Server SDK or N1QL mutation, and
+the mutations will then be automatically synchronized into the Couchbase Analytics Service.
diff --git a/content/analytics/7_using_index.html b/content/analytics/7_using_index.html index 5ff5e7c7ea..183679f81e 100644 --- a/content/analytics/7_using_index.html +++ b/content/analytics/7_using_index.html @@ -3,12 +3,11 @@ - Using Indexes
-

Using Indexes

+

Indexes can speedup both selection queries and join queries if they are applied properly. The following two sections describe @@ -42,7 +41,7 @@

Selection Queries

The following query uses the index because it has an equality predicate (=) on a field (c_s) that is a prefix of the -indexed key (c_s, c_d)

+indexed key (c_s, c_d):

SELECT f.c_x as res
 FROM foo f
@@ -50,6 +49,14 @@ 
Selection Queries
+

If you would like an available index to not be used for a particular query predicate (e.g., because there will be many, many matching objects), a skip-index hint can be used:

+ +
SELECT f.c_x as res
+FROM foo f
+WHERE f.c_s /*+ skip-index */ = 'world';
+
+
+

If multiple indexes are eligible access paths, there can be two cases:

    diff --git a/content/analytics/8_builtin.html b/content/analytics/8_builtin.html index 7dc8d9f863..819d0174be 100644 --- a/content/analytics/8_builtin.html +++ b/content/analytics/8_builtin.html @@ -3,12 +3,11 @@ - Builtin Functions
    -

    Builtin Functions

    +

    An instance of Couchbase Analytics data model can be a *primitive type* (BOOLEAN, STRING, BIGINT, or DOUBLE), a *special type* (NULL or MISSING), or a *composite type* (OBJECT or ARRAY).

    diff --git a/content/analytics/analytics.ditamap b/content/analytics/analytics.ditamap index 0ab0764bda..789cfe8ac5 100644 --- a/content/analytics/analytics.ditamap +++ b/content/analytics/analytics.ditamap @@ -2,33 +2,27 @@ Couchbase Analytics Service - - - - - - - - - + + - - - - + + + + - - - - - - - - - - - + + + + + + + + + + + + diff --git a/content/analytics/appendix_1_keywords.html b/content/analytics/appendix_1_keywords.html index 30a89a07aa..fcd525872b 100644 --- a/content/analytics/appendix_1_keywords.html +++ b/content/analytics/appendix_1_keywords.html @@ -3,12 +3,11 @@ - Appendix 1. Reserved keywords
    -

    Reserved keywords

    +

    All reserved keywords are listed in the following table:

    diff --git a/content/analytics/appendix_2_parameters.html b/content/analytics/appendix_2_parameters.html index c9709d4fa3..e0dc7bf3ac 100644 --- a/content/analytics/appendix_2_parameters.html +++ b/content/analytics/appendix_2_parameters.html @@ -2,13 +2,12 @@ - - Appendix 2. Performance Tuning +
    -

    Performance Tuning

    + + + +

    In this Appendix, we'll look at how variables are bound and how names are resolved. +Names can appear in every clause of a query. +Sometimes a name consists of just a single identifier, e.g., region or revenue. +More often a name will consist of two identifiers separated by a dot, e.g., customer.address. +Occasionally a name may have more than two identifiers, e.g., policy.owner.address.zipcode. +Resolving a name means determining exactly what the (possibly multi-part) name refers to. +It is necessary to have well-defined rules for how to resolve a name in cases of ambiguity. +(In the absence of schemas, such cases arise more commonly, and also differently, than they do in SQL.)

    + +

    The basic job of each clause in a query block is to bind variables. +Each clause sees the variables bound by previous clauses and may bind additional variables. +Names are always resolved with respect to the variables that are bound ("in scope") at the place where the name use in question occurs. +It is possible that the name resolution process will fail, which may lead to an empty result or an error message.

    + +

    One important bit of background: When the system is reading a query and resolving its names, it has a list of all the available dataverses and datasets. +As a result, it knows whether a.b is a valid name for dataset b in dataverse a. +However, the system does not in general have knowledge of the schemas of the data inside the datasets; remember that this is a much more open world. +As a result, in general the system cannot know whether any object in a particular dataset will have a field named c. +These assumptions affect how errors are handled. +If you try to access dataset a.b and no dataset by that name exists, you will get an error and your query will not run. +However, if you try to access a field c in a collection of objects, your query will run and return missing for each object that doesn't have a field named c – this is because it’s possible that some object (someday) could have such a field.

    + +

    Binding Variables

    + +

    Variables can be bound in the following ways:

    + +
      +
    1. +

      WITH and LET clauses bind a variable to the result of an expression in a straightforward way

      + +

      Examples:

      + +

      WITH cheap_parts AS (SELECT partno FROM parts WHERE price < 100) +binds the variable cheap_parts to the result of the subquery.

      + +

      LET pay = salary + bonus +binds the variable pay to the result of evaluating the expression salary + bonus.

      +
      2. +
    2. +

      FROM, GROUP BY, and SELECT clauses have optional AS subclauses that contain an expression and a name (called an iteration variable in a FROM clause, or an alias in GROUP BY or SELECT.)

      + +

      Examples:

      + +

      FROM customer AS c, order AS o

      + +

      GROUP BY salary + bonus AS total_pay

      + +

      SELECT MAX(price) AS highest_price

      + +

      An AS subclause always binds the name (as a variable) to the result of the expression (or, in the case of a FROM clause, to the individual members of the collection identified by the expression.)

      + +

      It's always a good practice to use the keyword AS when defining an alias or iteration variable. +However, as in SQL, the syntax allows the keyword AS to be omitted. +For example, the FROM clause above could have been written like this:

      + +

      FROM customer c, order o

      + +

      Omitting the keyword AS does not affect the binding of variables. +The FROM clause in this example binds variables c and o whether the keyword AS is used or not.

      + +

      In certain cases, a variable is automatically bound even if no alias or variable-name is specified. +Whenever an expression could have been followed by an AS subclause, if the expression consists of a simple name or a path expression, that expression binds a variable whose name is the same as the simple name or the last step in the path expression. +Here are some examples:

      + +

      FROM customer, order binds iteration variables named customer and order

      + +

      GROUP BY address.zipcode binds a variable named zipcode

      + +

      SELECT item[0].price binds a variable named price

      + +

      Note that a FROM clause iterates over a collection (usually a dataset), binding a variable to each member of the collection in turn. +The name of the collection remains in scope, but it is not a variable. +For example, consider this FROM clause used in a self-join:

      + +

      FROM customer AS c1, customer AS c2

      + +

      This FROM clause joins the customer dataset to itself, binding the iteration variables c1 and c2 to objects in the left-hand-side and right-hand-side of the join, respectively. +After the FROM clause, c1 and c2 are in scope as variables, and customer remains accessible as a dataset name but not as a variable.

      +
      3. +
    3. +

      Special rules for GROUP BY:

      + +
        +
      1. +

        If a GROUP BY clause specifies an expression that has no explicit alias, it binds a pseudo-variable that is lexicographically identical to the expression itself. +For example:

        + +

        GROUP BY salary + bonus binds a pseudo-variable named salary + bonus.

        + +

        This rule allows subsequent clauses to refer to the grouping expression (salary + bonus) even though its constituent variables (salary and bonus) are no longer in scope. +For example, the following query is valid:

        + + 
        FROM employee
+GROUP BY salary + bonus
+HAVING salary + bonus > 1000
+SELECT salary + bonus, COUNT(*) AS how_many
+
        +
        + +

        While it might have been more elegant to explicitly require an alias in cases like this, the pseudo-variable rule is retained for SQL compatibility. +Note that the expression salary + bonus is not actually evaluated in the HAVING and SELECT clauses (and could not be since salary and bonus are no longer individually in scope). +Instead, the expression salary + bonus is treated as a reference to the pseudo-variable defined in the GROUP BY clause.

        +
        2. +
      2. +

        A GROUP BY clause may be followed by a GROUP AS clause that binds a variable to the group. +The purpose of this variable is to make the individual objects inside the group visible to subqueries that may need to iterate over them.

        + +

        The GROUP AS variable is bound to a multiset of objects. +Each object represents one of the members of the group. +Since the group may have been formed from a join, each of the member-objects contains a nested object for each variable bound by the nearest FROM clause (and its LET subclause, if any). +These nested objects, in turn, contain the actual fields of the group-member. +To understand this process, consider the following query fragment:

        + + 
        FROM parts AS p, suppliers AS s
+WHERE p.suppno = s.suppno
+GROUP BY p.color GROUP AS g
+
        +
        + +

        Suppose that the objects in parts have fields partno, color, and suppno. +Suppose that the objects in suppliers have fields suppno and location.

        + +

        Then, for each group formed by the GROUP BY, the variable g will be bound to a multiset with the following structure:

        + + 
        [ { "p": { "partno": "p1", "color": "red", "suppno": "s1" },
+    "s": { "suppno": "s1", "location": "Denver" } },
+  { "p": { "partno": "p2", "color": "red", "suppno": "s2" },
+    "s": { "suppno": "s2", "location": "Atlanta" } },
+  ...
+]
+
        +
        +
        3. +
      +
      4. +
    + +

    Scoping

    + +

    In general, the variables that are in scope at a particular position are those variables that were bound earlier in the current query block, in outer (enclosing) query blocks, or in a WITH clause at the beginning of the query. +More specific rules follow.

    + +

    The clauses in a query block are conceptually processed in the following order:

    + +
      +
    • FROM (followed by LET subclause, if any)
      • +
    • WHERE
      • +
    • GROUP BY (followed by LET subclause, if any)
      • +
    • HAVING
      • +
    • SELECT or SELECT VALUE
      • +
    • ORDER BY
      • +
    • OFFSET
      • +
    • LIMIT
      • +
    + +

    During processing of each clause, the variables that are in scope are those variables that are bound in the following places:

    + +
      +
    1. +

      In earlier clauses of the same query block (as defined by the ordering given above).

      + +

      Example: FROM orders AS o SELECT o.date +The variable o in the SELECT clause is bound, in turn, to each object in the dataset orders.

      +
      2. +
    2. +

      In outer query blocks in which the current query block is nested. +In case of duplication, the innermost binding wins.

      +
      3. +
    3. +

      In the WITH clause (if any) at the beginning of the query.

      +
      4. +
    + +

    However, in a query block where a GROUP BY clause is present:

    + +
      +
    1. +

      In clauses processed before GROUP BY, scoping rules are the same as though no GROUP BY were present.

      +
      2. +
    2. +

      In clauses processed after GROUP BY, the variables bound in the nearest FROM-clause (and its LET subclause, if any) are removed from scope and replaced by the variables bound in the GROUP BY clause (and its LET subclause, if any). +However, this replacement does not apply inside the arguments of the five SQL special aggregating functions (MIN, MAX, AVG, SUM, and COUNT). +These functions still need to see the individual data items over which they are computing an aggregation. +For example, after FROM employee AS e GROUP BY deptno, it would not be valid to reference e.salary, but AVG(e.salary) would be valid.

      +
      3. +
    + +

    Special case: In an expression inside a FROM clause, a variable is in scope if it was bound in an earlier expression in the same FROM clause. +Example:

    + +
    FROM orders AS o, o.items AS i
+
    +
    + +

    The reason for this special case is to support iteration over nested collections.

    + +

    Note that, since the SELECT clause comes after the WHERE and GROUP BY clauses in conceptual processing order, any variables defined in SELECT are not visible in WHERE or GROUP BY. +Therefore the following query will not return what might be the expected result (since in the WHERE clause, pay will be interpreted as a field in the emp object rather than as the computed value salary + bonus):

    + +
    SELECT name, salary + bonus AS pay
+FROM emp
+WHERE pay > 1000
+ORDER BY pay
+
    +
    + +

    The likely intent of the query above can be accomplished as follows:

    + +
    FROM emp AS e
+LET pay = e.salary + e.bonus
+WHERE pay > 1000
+SELECT e.name, pay
+ORDER BY pay
+
    +
    + +

    Resolving Names

    + +

    The process of name resolution begins with the leftmost identifier in the name. +The rules for resolving the leftmost identifier are:

    + +
      +
    1. +

      In a FROM clause: Names in a FROM clause identify the collections over which the query block will iterate. +These collections may be stored datasets or may be the results of nested query blocks. +A stored dataset may be in a named dataverse or in the default dataverse. +Thus, if the two-part name a.b is in a FROM clause, a might represent a dataverse and b might represent a dataset in that dataverse. +Another example of a two-part name in a FROM clause is FROM orders AS o, o.items AS i. +In o.items, o represents an order object bound earlier in the FROM clause, and items represents the items object inside that order.

      + +

      The rules for resolving the leftmost identifier in a FROM clause (including a JOIN subclause), or in the expression following IN in a quantified predicate, are as follows:

      + +
        +
      1. +

        If the identifier matches a variable-name that is in scope, it resolves to the binding of that variable. +(Note that in the case of a subquery, an in-scope variable might have been bound in an outer query block; this is called a correlated subquery.)

        +
        2. +
      2. +

        Otherwise, if the identifier is the first part of a two-part name like a.b, the name is treated as dataverse.dataset. +If the identifier stands alone as a one-part name, it is treated as the name of a dataset in the default dataverse. +An error will result if the designated dataverse or dataset does not exist.

        +
        3. +
      +
      2. +
    2. +

      Elsewhere in a query block: In clauses other than FROM, a name typically identifies a field of some object. +For example, if the expression a.b is in a SELECT or WHERE clause, it's likely that a represents an object and b represents a field in that object.

      + +

      The rules for resolving the leftmost identifier in clauses other than the ones listed in Rule 1 are:

      + +
        +
      1. +

        If the identifier matches a variable-name that is in scope, it resolves to the binding of that variable. +(In the case of a correlated subquery, the in-scope variable might have been bound in an outer query block.)

        +
        2. +
      2. +

        (The "Single Variable Rule"): Otherwise, if the FROM clause (or a LET clause if there is no FROM clause) in the current query block binds exactly one variable, the identifier is treated as a field access on the object bound to that variable. +For example, in the query FROM customer SELECT address, the identifier address is treated as a field in the object bound to the variable customer. +At runtime, if the object bound to customer has no address field, the address expression will return missing. +If the FROM clause (and its LET subclause, if any) in the current query block binds multiple variables, name resolution fails with an "ambiguous name" error. +Note that the Single Variable Rule searches for bound variables only in the current query block, not in outer (containing) blocks. +The purpose of this rule is to permit the compiler to resolve field-references unambiguously without relying on any schema information.

        + +

        Exception: In a query that has a GROUP BY clause, the Single Variable Rule does not apply in any clauses that occur after the GROUP BY because, in these clauses, the variables bound by the FROM clause are no longer in scope. +In clauses after GROUP BY, only Rule 2.1 applies.

        +
        3. +
      +
      3. +
    3. +

      In an ORDER BY clause following a UNION ALL expression:

      + +

      The leftmost identifier is treated as a field-access on the objects that are generated by the UNION ALL. +For example:

      + + 
      query-block-1
+UNION ALL
+query-block-2
+ORDER BY salary
+
      +
      + +

      In the result of this query, objects that have a foo field will be ordered by the value of this field; objects that have no foo field will appear at at the beginning of the query result (in ascending order) or at the end (in descending order.)

      +
      4. +
    4. +

      Once the leftmost identifier has been resolved, the following dots and identifiers in the name (if any) are treated as a path expression that navigates to a field nested inside that object. +The name resolves to the field at the end of the path. +If this field does not exist, the value missing is returned.

      +
      5. +
    + +
    +
    +
    + + diff --git a/content/analytics/cbq.html b/content/analytics/cbq.html index f834cb5897..56a1ec891a 100644 --- a/content/analytics/cbq.html +++ b/content/analytics/cbq.html @@ -3,16 +3,15 @@ - cbq
    -

    Command Line Tool

    +

    You can use the command line tool, cbq, to run queries. cbq is the command line shell for executing queries against the Query service as well as the Analytics service in Couchbase. diff --git a/content/analytics/config.html b/content/analytics/config.html index ba1eb00430..9c22ba3a78 100644 --- a/content/analytics/config.html +++ b/content/analytics/config.html @@ -3,15 +3,14 @@ - Couchbase Analytics configuration parameters

    -

    Couchbase Analytics Service Configuration Parameters

    +
    Note: The configuration parameters for Analytics are @@ -29,17 +28,12 @@

    Couchbase Analytic

    The following parameters can be used with the Cluster Controller Configuration API to configure the Analytics Service cluster controller.

    - - - - - - - - - - - +
    ParameterDescriptionDefault
    + + + + + @@ -53,6 +47,11 @@

    Couchbase Analytic

    + + + + + @@ -61,12 +60,12 @@

    Couchbase Analytic

    - + - + @@ -99,17 +98,12 @@

    Couchbase Analytic

    The following parameters can be used with the Node Configuration API to configure the Analytics Service nodes.

    -

    ParameterDescriptionDefault
    The number of threads to service HTTP requests 16
    coresMultiplierThe factor to multiply by the number of cores to determine maximum query concurrent execution level3
    enforceFrameWriterProtocol A flag indicating if runtime should enforce frame writer protocol and detect bad behaving operators
    heartbeatMaxMisses Sets the maximum number of missed heartbeats before a node is marked as dead54
    heartbeatPeriod Sets the time duration between two heartbeats from each node controller in milliseconds1000030000
    jobHistorySize
    - - - - - - - - - - +
    ParameterDescriptionDefault
    + + + + + @@ -158,11 +152,6 @@

    Couchbase Analytic

    - - - - - @@ -243,21 +232,21 @@

    Couchbase Analytic

    + + + + + - + - - - - - @@ -266,7 +255,7 @@

    Couchbase Analytic

    - +
    ParameterDescriptionDefault
    The memory budget (in bytes) for a sort operator instance in a partition 33554432 (32 MB)
    instanceNameThe name of this cluster instanceDEFAULT_INSTANCE
    iodevices (read-only) Comma separated list of IO Device mount pointsThe maximum acceptable false positive rate for bloom filters associated with LSM indexes 0.01
    storageMaxActiveWritableDatasetsThe maximum number of datasets that can be concurrently modified8
    storageMemorycomponentGlobalbudget The size of memory allocated to the memory components. The value should be a multiple of the memory component page size715915264 (682.75 MB)1/4 of the allocated Analytics Service memory
    storageMemorycomponentNumcomponents The number of memory components to be used per lsm index 2
    storageMemorycomponentNumpagesThe number of pages to allocate for a memory component. This budget is shared by all the memory components of the primary index and all its secondary indexes across all I/O devices on a node. Note: in-memory components usually has fill factor of 75% since the pages are 75% full and the remaining 25% is un-utilized1/16th of the storageMemorycomponentGlobalbudget value
    storageMemorycomponentPagesize The page size in bytes for pages allocated to memory components
    storageMetadataMemorycomponentNumpages The number of pages to allocate for a metadata memory component1/64th of the storageMemorycomponentGlobalbudget value or 256, whichever is larger8
    diff --git a/content/analytics/html-generation/_config.yml b/content/analytics/html-generation/_config.yml index 1172e0e5b7..7afb6a3373 100644 --- a/content/analytics/html-generation/_config.yml +++ b/content/analytics/html-generation/_config.yml @@ -26,6 +26,8 @@ github_username: jekyll # Build settings markdown: kramdown +kramdown: + smart_quotes: ["apos", "apos", "quot", "quot"] theme: minima exclude: - Gemfile diff --git a/content/analytics/html-generation/_layouts/default.html b/content/analytics/html-generation/_layouts/default.html index fd349520e3..9297ae6aab 100644 --- a/content/analytics/html-generation/_layouts/default.html +++ b/content/analytics/html-generation/_layouts/default.html @@ -3,7 +3,6 @@ - {{ page.title }}
    diff --git a/content/analytics/html-generation/build.sh b/content/analytics/html-generation/build.sh new file mode 100755 index 0000000000..b518679684 --- /dev/null +++ b/content/analytics/html-generation/build.sh @@ -0,0 +1,46 @@ +#!/bin/bash + +DOC_PROJECT=$1 # should be /analytics/cbas/doc + +[ -f $DOC_PROJECT/pom.xml ] || { + echo "$DOC_PROJECT not valid (does not contain a pom.xml)" + exit 1 +} + +for FILE in Gemfile _config.yml +do + [ -f $FILE ] || { + echo required file $FILE missing + exit 1 + } +done + +MD_DIR=$DOC_PROJECT/target/markdown/ + +echo '*** copy files from' $MD_DIR + +rsync -r $MD_DIR . + +echo '*** clean _site' + +rm -rf _site/* + +echo '*** run jekyll' + +bundle exec jekyll b + +echo '*** fix HTML ***' + +for FILE in $(find . -name "*.html") +do + # remove the first

    tag in the body + awk 'NR==1,//{sub(//, "")} 1' $FILE > tmp + #echo "******** $FILE" + mv tmp $FILE +done + +echo '*** move generated files' + +rsync -r _site/* .. + +git st diff --git a/content/analytics/images/workbench-results.png b/content/analytics/images/workbench-results.png index 9a9f15fd32..eec78b3419 100644 Binary files a/content/analytics/images/workbench-results.png and b/content/analytics/images/workbench-results.png differ diff --git a/content/analytics/initialize-cluster.html b/content/analytics/initialize-cluster.html index 21c7642092..14c0efd0e8 100644 --- a/content/analytics/initialize-cluster.html +++ b/content/analytics/initialize-cluster.html @@ -3,13 +3,12 @@ - Initialize the Cluster

    To initialize a Couchbase Cluster with the Analytics Service, the instructions to diff --git a/content/analytics/introduction.html b/content/analytics/introduction.html index 3dd0efe514..ad21c30f2f 100644 --- a/content/analytics/introduction.html +++ b/content/analytics/introduction.html @@ -3,40 +3,38 @@ - Introduction

    -

    Introduction

    +

    Couchbase Analytics is a Developer Preview of a parallel data management capability for Couchbase Server. Couchbase Analytics is designed to efficiently run complex queries over many records. -By complex queries, we mean large ad hoc join, set, aggregation, and grouping operations, -any of which may result in long running queries, high CPU usage, high memory consumption, -and excessive network latency in data fetching and cross node coordination.

    +By complex queries, we mean large ad hoc join, set, aggregation, and grouping operations, any of which may result in +long running queries, high CPU usage, high memory consumption, and excessive network latency in data fetching and +cross node coordination.

    -

    Regardless of the technology used, analytic queries can be predetermined or ad hoc, -and can be cheap or expensive depending on how much data processing they need. -Performance challenges can arise when queries access large numbers of documents and when -queries are not supported by a secondary index, as often happens with ad hoc analytics -such as you may perform using a visualization tool.

    +

    Regardless of the technology used, analytic queries can be predetermined or ad hoc, and can be cheap or expensive +depending on how much data processing they need. +Performance challenges can arise when queries access large numbers of documents and when queries are not supported by +a secondary index, as often happens with ad hoc analytics such as you may perform using a visualization tool.

    -

    Couchbase Analytics is designed to support truly ad hoc queries in a reasonable amount of time, -even when scans are required. Because Analytics supports efficient parallel query processing and bulk data handling, -Couchbase Analytics is still preferred for expensive queries, even when those queries are predetermined -and might therefore be supported by an index.

    +

    Couchbase Analytics is designed to support truly ad hoc queries in a reasonable amount of time, even when scans are +required. Because Analytics supports efficient parallel query processing and bulk data handling, Couchbase Analytics +is still preferred for expensive queries, even when those queries are predetermined and might therefore be supported +by an index.

    The Couchbase Analytics approach has significant advantages compared to alternatives:

    • -

      Common data model: Couchbase Analytics natively supports the same rich, flexible-schema document -data model used in Couchbase Server, rather than trying to force your data into an RDBMS model.

      +

      Common data model: Couchbase Analytics natively supports the same rich, flexible-schema document data model +used in Couchbase Server, rather than trying to force your data into an RDBMS model.

    • Workload isolation: Operational query latency and throughput are protected from slow downs due to analytical @@ -44,28 +42,31 @@

      Introduction

    • High data freshness: Couchbase Analytics uses -DCP, -a fast memory-to-memory protocol that Couchbase Server -nodes use to synchronize data among themselves. Because of this, Analytics runs on data that’s extremely current, -without extract, transform, load (ETL) or other hassles and delays.

      +DCP, +a fast memory-to-memory protocol that Couchbase Server nodes use to synchronize data among themselves. +Because of this, Analytics runs on data that’s extremely current, without extract, transform, load (ETL) or other +hassles and delays.

    SQL++ Query Language

    Couchbase Analytics is programmed using the SQL++ query language, which is a next-generation declarative query language. -SQL++ has much in common with SQL, but it also includes a small number of extensions that address the different -data models the two languages were designed to serve. Compared to SQL, SQL++ is much newer and targets the nested, +SQL++ has much in common with SQL, but it also includes a small number of extensions that address the different data +models the two languages were designed to serve. Compared to SQL, SQL++ is much newer and targets the nested, schema-optional or even schema-less world of modern NoSQL systems.

    -

    Now, you may wonder why Couchbase Analytics uses a query language other than N1QL, the query language used by -the Couchbase Server’s query service. The answer is, this is a temporary situation. Both SQL++ and N1QL are -quite similar; in fact N1QL was inspired by the open language specification SQL++ and is a proper subset of -that specification. In the long term, the two query languages will merge so that Couchbase Server can be queried -using one single query language.

    +

    Now, you may wonder why Couchbase Analytics uses a query language other than N1QL, the query language used by Couchbase +Server’s query service. In fact, both SQL++ and N1QL are quite similar, with SQL++ offering some advancements beyond +N1QL. The section Differences from N1QL provides additional details.

    For more details, refer to the SQL++ Language Reference section.

    +

    What's Next

    + +

    Now you can continue to Running Your First Analytics Query +to get familiar with the ways you can run a query or continue directly to the Tutorial.

    +
    diff --git a/content/analytics/primer-beer.html b/content/analytics/primer-beer.html index b32b0eb2e9..b3eb4b5559 100644 --- a/content/analytics/primer-beer.html +++ b/content/analytics/primer-beer.html @@ -3,68 +3,81 @@ - Tutorial
    -

    Tutorial

    +

    + +

    This tutorial introduces the main features of the new Couchbase Analytics through examples.

    Welcome to the Analytics DP!

    -

    Since few things in life are more important than choosing the right beer to pair with one’s main dish, +

    Since few things in life are more important than choosing the right beer to pair with one's main dish, this tutorial explores the pairing of Analytics and beers. -You’ll create a set of Analytics shadow datasets based on the Couchbase beer-sample bucket -and then explore a set of illustrative queries as a quick way to get familiar with the new Analytics user experience. -The complete set of steps to create and connect the sample datasets are included, along with a collection of runnable SQL++ -queries and their expected results.

    +You'll create a set of Analytics datasets based on the Couchbase beer-sample bucket and then explore a set of +illustrative queries as a quick way to get familiar with the new Analytics user experience. +The complete set of steps to create and connect the sample datasets are included, along with a collection of runnable +SQL++ queries and their expected results.

    -

    To run these examples, you must complete the set up as described in Quick Start.

    +

    As you read through this document, you should try each step for yourself on your own Analytics instance. +You can use your favorite Analytics interface to do this: the Analytics Workbench, the cbq shell, +or the REST API. For details, see Runninq Queries.

    -

    You can verify that everything’s working by issuing a simple SQL++ test query as shown below:

    +

    You can verify that everything's working by issuing a simple SQL++ test query as shown below:

    "Let there be beer!";
    -

    As you read through this document, you should try each step for yourself on your own Analytics instance. -You can use your favorite Analytics interface to do this: the Analytics Workbench, the modified version of cbq that ships with Analytics, or the REST API. For details, see Runninq Queries. -Once you have reached the end of this tutorial, you will be armed and dangerous, -having all the basic Analytics knowledge that you’ll need to start down the path of exploring the power of NoSQL data analytics. +

    Once you have reached the end of this tutorial, you will be armed and dangerous, having all the basic Analytics +knowledge that you'll need to start down the path of exploring the power of NoSQL data analytics. You will find this to be a freeing experience. -Analytics SQL++ queries never touch your Couchbase data servers, running instead (in parallel) on real-time shadow copies of your data. -As a result, you’ll find yourself in a world where you can ask the system anything! - you won’t need to worry about -slowing down the Couchbase Server nodes with complex queries, you won’t have to create indexes to explore data, -and the Analytics query language (SQL++) won’t try to keep you from asking queries that would be too performance-costly.

    +Analytics SQL++ queries never touch your Couchbase data servers, running instead (in parallel) on real-time shadow +copies of your data. +As a result, you'll find yourself in a world where you can ask the system anything! - you won't need to worry about +slowing down the Couchbase Server nodes with complex queries, you won't have to create indexes to explore data, +and the Analytics query language (SQL++) won't try to keep you from asking queries that would be too performance-costly.

    The World of Data in Analytics

    +

    In this section you will learn about the Analytics world model for data.

    Organizing Data in Analytics

    -

    The top-level organizing concept in the Analytics data world is the dataverse. A dataverse, short for data universe, -is a namespace that gives you a place to create and manage datasets and other artifacts for a given Analytics -application. In that respect, a dataverse is similar to a database or a schema in a relational DBMS. To store your -data in Analytics, you create a dataverse and then use it to hold the datasets for your own data. -We’ll get to this next. You get a Default dataverse for free, and Analytics will just use that if you don’t specify -another dataverse.

    - -

    Datasets are containers that hold collections of JSON objects. They are similar to tables in an RDBMS or keyspaces -in N1QL. In this tutorial, all of the datasets you create are shadow datasets that contain real-time synchronized -copies of selected data from Couchbase Server. For shadow datasets, Analytics uses DCP to automatically maintain -its own local representations for the JSON documents in the Couchbase Server data nodes.

    +

    The top-level organizing concept in the Analytics data world is the dataverse. +A dataverse, short for data universe, is a namespace that gives you a place to create and manage datasets and +other artifacts for a given Analytics application. +In that respect, a dataverse is similar to a database or a schema in a relational DBMS. +To store your data in Analytics, you create a dataverse and then use it to hold the datasets for your own data. +We'll get to this next. +You get a Default dataverse for free, and Analytics will just use that if you don't specify another dataverse.

    + +

    Datasets are containers that hold collections of JSON objects. +They are similar to tables in an RDBMS or keyspaces in N1QL. +In this tutorial, all of the datasets you create contain real-time synchronized copies of selected data from +Couchbase Server. +Analytics uses DCP to automatically maintain its own local representations for the JSON documents in the Couchbase +Server data nodes.

    -

    OK, let’s put these concepts to work.

    +

    OK, let's put these concepts to work.

    -

    A newly created Analytics instance starts out empty. That is, it contains no data other than the Analytics system -catalogs. These system catalogs live in a special dataverse called the Metadata dataverse. If you want to know what -dataverses have been defined so far, you can query the Dataverse dataset in the Metadata dataverse as shown below:

    +

    A newly created Analytics instance starts out empty. +That is, it contains no data other than the Analytics system catalogs. +These system catalogs live in a special dataverse called the Metadata dataverse. +If you want to know what dataverses have been defined so far, you can query the Dataverse dataset in the Metadata +dataverse as shown below:

    SELECT * FROM Metadata.`Dataverse`;
    @@ -95,32 +108,33 @@

    Organ

    The output above shows that a fresh Analytics instance starts out with two dataverses, one called Metadata (the system catalog) and one called Default (available for holding data). -This sample scenario deals with the information about beers and breweries, which you’ll need to get from -Couchbase Server. To keep things simple, we will use the Default dataverse here. That means your first task is to tell -Analytics about the Couchbase Server data that you want to shadow and the shadow datasets where you want it to live. +This sample scenario deals with the information about beers and breweries, which you'll need to get from +Couchbase Server. +To keep things simple, we will use the Default dataverse here. +That means your first task is to tell Analytics about the Couchbase Server data that you want to shadow and the +datasets where you want it to live. The following Analytics DDL statements show you how to tell Analytics about the beer-sample bucket in Couchbase Server, where all of the beer and brewery information resides, and ask Analytics to shadow the data using two datasets, breweries and beers.

    CREATE BUCKET beerBucket WITH {"name":"beer-sample"};
-CREATE SHADOW DATASET breweries ON beerBucket WHERE `type` = "brewery";
-CREATE SHADOW DATASET beers ON beerBucket WHERE `type` = "beer";
+CREATE DATASET breweries ON beerBucket WHERE `type` = "brewery";
+CREATE DATASET beers ON beerBucket WHERE `type` = "beer";

    The first statement (CREATE BUCKET) gives Analytics the information needed to access the data of interest from Couchbase Server. -The next two statements (CREATE SHADOW DATASET) create the target datasets in Analytics for the information of -interest. +The next two statements (CREATE DATASET) create the target datasets in Analytics for the information of interest. Notice how WHERE clauses are utilized to direct beer-related data into separate, type-specific datasets for easier querying. -Each of these datasets will be hash-partitioned (sharded) across all of the nodes running instances of -the Analytics service. +Each of these datasets will be hash-partitioned (sharded) across all of the nodes running instances of the Analytics +service. Hash partitioning sets the stage for the parallel processing that Analytics employs when processing analytical queries.

    -

    To actually initiate the shadowing relationship of these datasets to the data in Couchbase Server, -one more step is needed, namely:

    +

    To actually initiate the shadowing relationship of these datasets to the data in Couchbase Server, one more step is +needed, namely:

    CONNECT BUCKET beerBucket;
    @@ -129,10 +143,10 @@

    Organ

    Once you run this statement, Analytics begins making its copy of the Couchbase Server documents and continuously monitors it for changes.

    -

    What’s Lurking in the Shadows?

    +

    What's Lurking in the Shadows?

    -

    Next, verify that your shadow datasets are really there and being populated. The following SQL++ -SELECT statements are one good way to accomplish that:

    +

    Next, verify that your datasets are really there and being populated. +The following SQL++ SELECT statements are one good way to accomplish that:

    SELECT ds.BucketName, ds.DatasetName, ds.`Filter`
 FROM Metadata.`Dataset` ds
@@ -140,8 +154,8 @@ 
    What
    -

    The query looks in the Analytics system catalogs for datasets that have been created in the Default -dataverse. At this point, assuming that you are just getting started, you will see only your two new shadow datasets:

    +

    The query looks in the Analytics system catalogs for datasets that have been created in the Default dataverse. +At this point, assuming that you are just getting started, you will see only your two new datasets:

    [ {
     "DatasetName" : "beers",
@@ -249,15 +263,16 @@ 
    What
 
    SQL++: Querying Your Analytics Data
    
 
 
    Congratulations! You now have your Couchbase Server beer-related data being shadowed in Analytics.
-You’re ready to start running ad hoc queries against your breweries and beers datasets.
    
+You're ready to start running ad hoc queries against your breweries and beers datasets.
    
 
-
    To do this, you’ll program Analytics using the SQL++ query language, a SQL-inspired language
-designed for working with semistructured data. SQL++ has much in common with SQL, but there are
-differences due to the data model that SQL++ is designed to serve. SQL was designed in the 1970’s
-to interact with the flat, schema-ified world of relational databases.
+
    To do this, you'll program Analytics using the SQL++ query language, a SQL-inspired language
+designed for working with semistructured data.
+SQL++ has much in common with SQL, but there are differences due to the data model that SQL++ is designed to serve.
+SQL was designed in the 1970's to interact with the flat, schema-ified world of relational databases.
 SQL++ is designed for the nested, schema-less (or schema-optional, in Analytics) world of NoSQL systems.
 SQL++ offers a mostly familiar paradigm for experienced SQL users to use to query and manipulate data
-in Analytics. SQL++ is also related to N1QL, the current query language used in Couchbase Server.
+in Analytics.
+SQL++ is also related to N1QL, the current query language used in Couchbase Server.
 SQL++ is really a functional superset of N1QL that is a bit closer to SQL,
 and the differences between N1QL and SQL++ will eventually disappear in the future releases.
    
 
@@ -267,19 +282,20 @@ 
    SQL++ Language Reference
 and the list of built-in functions in the Function Reference.
-As you will learn, SQL++ is a highly composable expression language. Even the very simple expression
-1 + 1 is a valid SQL++ query that evaluates to
+As you will learn, SQL++ is a highly composable expression language.
+Even the very simple expression 1 + 1 is a valid SQL++ query that evaluates to
 2.
 Try it for yourself!
    
 
-
    It’s worth noting that each time you execute a SQL++ query, the Analytics query engine employs state-of-the-art
+
    It's worth noting that each time you execute a SQL++ query, the Analytics query engine employs state-of-the-art
 parallel processing algorithms similar to those used by the parallel relational DBMSs that power many
 enterprise data warehouses. Unlike those systems, Analytics also works on rich, flexible schema data.
    
 
-
    Let’s go ahead and write some queries and start learning SQL++ through examples.
    
+
    Let's go ahead and write some queries and start learning SQL++ through examples.
    
 
 
    Query 0 - Key-Based Lookup
    
-
    For your first query, let’s find a particular brewery based on its Couchbase Server key.
+
+
    For your first query, let's find a particular brewery based on its Couchbase Server key.
 You can do this for the Kona Brewing company as follows:
    
 
 
    SELECT meta(bw) AS meta, bw AS data
@@ -288,12 +304,12 @@ 
    Query 0 - K
 
    
 
    
 
-
    As in SQL, the query’s FROM clause binds the variable bw incrementally
-to the data instances residing in the
-dataset named breweries. Its WHERE clause selects only those bindings having the primary key of interest; the key
-is accessed (as in N1QL) by using the meta function to get to the meta-information about the objects. The SELECT
-clause returns all of the meta-information plus the data value (the selected brewery object in this case) for
-each binding that satisfies the predicate.
    
+
    As in SQL, the query's FROM clause binds the variable bw incrementally
+to the data instances residing in the dataset named breweries.
+Its WHERE clause selects only those bindings having the primary key of interest; the key is accessed (as in N1QL)
+by using the meta function to get to the meta-information about the objects.
+The SELECT clause returns all of the meta-information plus the data value (the selected brewery object in this
+case) for each binding that satisfies the predicate.
    
 
 
    The expected result for this query is as follows:
    
 
@@ -334,8 +350,10 @@ 
    Query 0 - K
 
    Notice how the resulting object of interest has two fields whose names were requested in the SELECT clause.
    
 
 
    Query 1 - Exact-Match Lookup
    
-
    The SQL++ language, like SQL, supports a variety of different predicates. For the next query, let’s find the same
-brewery information but in a slightly simpler or cleaner way based only on the data:
    
+
+
    The SQL++ language, like SQL, supports a variety of different predicates.
+For the next query, let's find the same brewery information but in a slightly simpler or cleaner way based only on
+the data:
    
 
 
    SELECT VALUE bw
 FROM breweries bw
@@ -343,7 +361,7 @@ 
    Query 1 - Exac
 
    
 
    
 
-
    This query’s expected result is:
    
+
    This query's expected result is:
    
 
 
    [
   {
@@ -371,10 +389,10 @@ 
    Query 1 - Exac
 
    
 
 
    In SQL++, you can select a single value (whether it be an atomic or scalar value or a object value or an array value)
-by using a SELECT VALUE clause as shown above. If you instead use the more SQL-familier
-SELECT clause, SQL++ will return objects instead of values, and since a given query may SELECT multiple values in
-its result (like our first query did), you get a slightly differently shaped result using
-SELECT bw instead of
+by using a SELECT VALUE clause as shown above.
+If you instead use the more SQL-familier SELECT clause, SQL++ will return objects instead of values, and since a
+given query may SELECT multiple values in its result (like our first query did), you get a slightly differently
+shaped result using SELECT bw instead of
 SELECT VALUE bw:
    
 
 
    [
@@ -405,6 +423,7 @@ 
    Query 1 - Exac
 
    
 
 
    Query 2 - Other Query Filters
    
+
 
    SQL++ can apply ranges and other conditions on any data type that supports the appropriate set of comparators.
 As an example, the next query applies a range condition together with a string condition to select breweries:
    
 
@@ -522,11 +541,12 @@ 
    Query

    Query 3 (and friends) - Equijoin

    -

    In addition to simply binding variables to data instances and returning them whole, -a SQL++ query can construct new objects to return based on combinations of variable bindings. + +

    In addition to simply binding variables to data instances and returning them whole, a SQL++ query can construct new +objects to return based on combinations of variable bindings. This gives SQL++ the power to do projections and joins much like those done using multi-table FROM clauses in SQL. -For example, suppose that you wanted a list of all breweries paired with their associated beers, -with the list enumerating the brewery name and the beer name for each such pair. +For example, suppose that you wanted a list of all breweries paired with their associated beers, with the list +enumerating the brewery name and the beer name for each such pair. You can do this as follows in SQL++, while also limiting the answer set size to at most 3 results:

    SELECT bw.name AS brewer, br.name AS beer
@@ -538,10 +558,10 @@ 
    Query 3 (and friends) -

    The result of this query is a sequence of new objects, one for each brewery/beer pair. -Each instance in the result will be a object containing two fields, "brewer" and "beer", -containing the brewery’s name and the beer’s name, respectively, for each brewery/beer pair. -Notice how the use of a SQL-style SELECT clause, as opposed to the new SQL++ SELECT VALUE -clause, automatically results in the construction of a new object value for each result.

    +Each instance in the result will be a object containing two fields, "brewer" and "beer", containing the brewery's +name and the beer's name, respectively, for each brewery/beer pair. +Notice how the use of a SQL-style SELECT clause, as opposed to the new SQL++ SELECT VALUE clause, automatically +results in the construction of a new object value for each result.

    The expected result of this example SQL++ join query is:

    @@ -563,7 +583,7 @@

    Query 3 (and friends) -

    If we were feeling lazy, e.g., while browsing our data casually, we might use SELECT * in SQL++ to -return all of the matching user/message data:

    +return all of the matching beer/brewery data:

    SELECT *
 FROM breweries bw, beers br
@@ -607,7 +627,7 @@ 
    Query 3 (and friends) -
       "abv": 6,
       "brewery_id": "512_brewing_company",
       "category": "German Ale",
-      "description": "(512) ALT is a German-style amber ale that is fermented cooler than typical ales and cold conditioned like a lager. ALT means "old" in German and refers to a beer style made using ale yeast after many German brewers had switched to newly discovered lager yeast. This ale has a very smooth, yet pronounced, hop bitterness with a malty backbone and a characteristic German yeast character. Made with 98% Organic 2-row and Munch malts and US noble hops.",
+      "description": "(512) ALT is a German-style amber ale that is fermented cooler than typical ales and cold conditioned like a lager. ALT means “old” in German and refers to a beer style made using ale yeast after many German brewers had switched to newly discovered lager yeast. This ale has a very smooth, yet pronounced, hop bitterness with a malty backbone and a characteristic German yeast character. Made with 98% Organic 2-row and Munch malts and US noble hops.",
       "ibu": 0,
       "name": "(512) ALT",
       "srm": 0,
@@ -691,8 +711,8 @@ 
    Query 3 (and friends) -
    -

    Die-hard SQL JOIN clause syntax fans will be happy to know that SQL++ hasn’t forgotten them, -so a result identical to the one immediately above can also be produced as follows:

    +

    Die-hard SQL JOIN clause syntax fans will be happy to know that SQL++ hasn't forgotten them, so a result identical +to the one immediately above can also be produced as follows:

    SELECT *
 FROM breweries bw JOIN beers br ON br.brewery_id = meta(bw).id
@@ -711,51 +731,52 @@ 
    Query 3 (and friends) -
    -

    This version of the query uses an explicit object constructor to build each result object. Note that the string field -names "bw" and "br" in the object constructor above are both simple SQL++ expressions themselves, so in the most -general case, even the resulting field names can be computed as part of the query, making SQL++ a very powerful tool -for slicing and dicing semistructured data.

    +

    This version of the query uses an explicit object constructor to build each result object. +Note that the string field names "bw" and "br" in the object constructor above are both simple SQL++ expressions +themselves, so in the most general case, even the resulting field names can be computed as part of the query, making +SQL++ a very powerful tool for slicing and dicing semistructured data.

    -

    (It is worth knowing, with respect to influencing Analytics’ query evaluation, that FROM and JOIN clauses - +

    (It is worth knowing, with respect to influencing Analytics' query evaluation, that FROM and JOIN clauses - also known as joins - are currently evaluated in order, with the left clause probing the data of the right clause.)

    Query 4 - Nested Outer Join

    -

    In order to support joins between tables with missing or dangling join tuples, the designers of SQL ended -up shoe-horning a subset of the relational algebra into SQL’s FROM clause syntax and providing a -variety of join types there for users to choose from (which SQL++ supports for SQL compatibility). -Left outer joins are particularly important in SQL, for example, to print a summary of customers and orders, -grouped by customer, without omitting those customers who haven’t placed any orders yet.

    - -

    The SQL++ language supports nesting, both of queries and of query results, -and the combination allows for a cleaner and more natural approach to such queries. -As an example, suppose you wanted for each brewery to produce an object that contains -the brewery name along with a list of all of the brewery’s offered beer names and alcohol percentages. + +

    In order to support joins between tables with missing or dangling join tuples, the designers of SQL ended up +shoe-horning a subset of the relational algebra into SQL's FROM clause syntax and providing a variety of join types +there for users to choose from (which SQL++ supports for SQL compatibility). +Left outer joins are particularly important in SQL, for example, to print a summary of customers and orders, grouped +by customer, without omitting those customers who haven't placed any orders yet.

    + +

    The SQL++ language supports nesting, both of queries and of query results, and the combination allows for a cleaner +and more natural approach to such queries. +As an example, suppose you wanted for each brewery to produce an object that contains the brewery name along with a +list of all of the brewery's offered beer names and alcohol percentages. In the flat (also known as 1NF) world of SQL, approximating this query would involve a left outer join between -breweries and beers, ordered by brewery, with the brewery name being repeated along side each beer’s information. +breweries and beers, ordered by brewery, with the brewery name being repeated along side each beer's information. In the richer (NoSQL) world of SQL++, this use case can be handled more naturally as follows:

    SELECT bw.name AS brewer,
   (SELECT br.name, br.abv FROM beers br
-   WHERE br.brewery_id = meta(bw).id) AS beers
+   WHERE br.brewery_id = meta(bw).id
+   ORDER BY br.name) AS beers
 FROM breweries bw
 ORDER BY bw.name
 LIMIT 2;
    -

    This SQL++ query binds the variable bw to the objects in breweries; -for each brewery, it constructs a result object containing a "brewer" field with the brewery’s name plus a "beers" -field with a nested collection of objects containing the beer name and alcohol percentage for each of the brewery’s -beers. +

    This SQL++ query binds the variable bw to the objects in breweries; for each +brewery, it constructs a result object containing a "brewer" field with the brewery's name plus a "beers" field with +a nested collection of objects containing the beer name and alcohol percentage for each of the brewery's beers. The nested collection field for each brewery is created using a correlated subquery.

    -Note: While it looks like nested loops could be involved in computing the result, -Analytics recognizes the equivalence of such a query to an outerjoin, so it will use an -efficient parallel join strategy when actually computing the query's result. +Note: While it looks like nested loops could be involved in computing the result, Analytics +recognizes the equivalence of such a query to an outerjoin, so it will use an efficient parallel join strategy when +actually computing the query's result.
    -

    Below is this example query’s expected output:

    +

    Below is this example query's expected output:

    [
   {
@@ -849,16 +870,17 @@ 
    Query 4 - Nested Outer Join

    Query 5 - Theta Join

    +

    Not all joins are expressible as equijoins and computable using equijoin-oriented algorithms. -The join predicates for some use cases involve predicates with functions; Analytics supports the -expression of such queries and will still evaluate them as best it can using nested loop based -techniques (and broadcast joins in the parallel case).

    +The join predicates for some use cases involve predicates with functions; Analytics supports the expression of such +queries and will still evaluate them as best it can using nested loop based techniques (and broadcast joins in the +parallel case).

    -

    As an example of such a use case, suppose that you wanted for each Arizona brewery to get -the brewery’s name, location, and a list of competitors’ names – where competitors are other -breweries that are geographically close to their location. -In SQL++, this can be accomplished in a manner similar to the previous query, but with locality plus -name inequality instead of a simple key equality condition in the correlated query’s WHERE clause:

    +

    As an example of such a use case, suppose that you wanted for each Arizona brewery to get the brewery's name, +location, and a list of competitors' names – where competitors are other breweries that are geographically close to +their location. +In SQL++, this can be accomplished in a manner similar to the previous query, but with locality plus name inequality +instead of a simple key equality condition in the correlated query's WHERE clause:

    SELECT bw1.name AS brewer, bw1.geo AS location,
   (SELECT VALUE bw2.name FROM breweries bw2
@@ -914,14 +936,15 @@ 
    Query 5 - Theta Join

    Query 6 - Existential Quantification

    +

    The expressive power of SQL++ includes support for queries involving some (existentially quantified) and all (universally quantified) query semantics. -Quantified predicates are especially useful for querying datasets involving nested collections of objects, -in order to find objects where some or all of their nested sets’ objects satisfy a condition of interest. -To illustrate their use in such situations, we start here by using another (orthogonal) feature of SQL++, -its WITH clause, to create a temporarily nested view of breweries and their beers. We then use an -existential (SOME) predicate to find those breweries whose beers include at least one IPA and return the -brewery’s name, phone number, and complete list of beer names and associated alcohol levels. +Quantified predicates are especially useful for querying datasets involving nested collections of objects, in order +to find objects where some or all of their nested sets' objects satisfy a condition of interest. +To illustrate their use in such situations, we start here by using another (orthogonal) feature of SQL++, its WITH +clause, to create a temporarily nested view of breweries and their beers. +We then use an existential (SOME) predicate to find those breweries whose beers include at least one IPA and return +the brewery's name, phone number, and complete list of beer names and associated alcohol levels. Here is the resulting SQL++ query:

    WITH nested_breweries AS
@@ -1037,7 +1060,9 @@ 
    Query 6 - Existent

    Query 7 - Universal Quantification

    -

    As an example of a universally quantified SQL++ query, we can find those breweries that only have seriously strong beers:

    + +

    As an example of a universally quantified SQL++ query, we can find those breweries that only have seriously strong +beers:

    WITH nested_breweries AS
  (
@@ -1045,20 +1070,21 @@ 
    Query 7 - Universal
         (
          SELECT br.name, br.abv FROM beers br
          WHERE br.brewery_id = meta(bw).id
-        ) AS beers
+         ORDER BY br.name) AS beers
   FROM breweries bw
  )
 SELECT VALUE nb FROM nested_breweries nb
 WHERE (EVERY b IN nb.beers SATISFIES b.abv >= 10)
   AND ARRAY_COUNT(nb.beers) > 0
-ORDER BY nb.brewer;
+ORDER BY nb.brewer
 LIMIT 5;

    Notice how the brewery predicate also makes sure that the set of beers for each qualifying brewery is non-empty; this is needed if we want our results to exclude breweries that currently offer no beers, as an empty set trivially -satifies a universal predicate. The expected result in this case is:

    +satifies a universal predicate. +The expected result in this case is:

    [
   {
@@ -1095,13 +1121,13 @@ 
    Query 7 - Universal
         "name": "Malheur Black Chocolate 2003",
         "abv": 12
       },
-      {
-        "name": "Malheur MM",
-        "abv": 10
-      },
       {
         "name": "Malheur Brut Reserve",
         "abv": 11
+      },
+      {
+        "name": "Malheur MM",
+        "abv": 10
       }
     ],
     "brewer": "Brouwerij De Landtsheer",
@@ -1132,6 +1158,7 @@ 
    Query 7 - Universal

    Query 8 - Simple Aggregation

    +

    Like SQL, the SQL++ language of Analytics provides support for computing aggregates over large amounts of data. As a very simple example, the following SQL++ query computes the total number of beers in a SQL-like way:

    @@ -1139,7 +1166,7 @@

    Query 8 - Simple Aggregati

    -

    This query’s result will be:

    +

    This query's result will be:

    [ {
     "num_beers" : 5891
@@ -1171,34 +1198,34 @@ 
    Query 8 - Simple Aggregati
 
    For each traditional SQL aggregate function F, SQL++ has a corresponding function ARRAY_F that
 can be used to perform the desired aggregate calculation.
 Each such function is a regular function that takes a collection-valued argument to aggregate over.
-Thus, the query above counts the results produced by the beer selection subquery, and the previous,
-more SQL-like versions are just syntactic sugar for SQL++ queries that logically use ARRAY_COUNT.
    
+Thus, the query above counts the results produced by the beer selection subquery, and the previous, more SQL-like
+versions are just syntactic sugar for SQL++ queries that logically use ARRAY_COUNT.
    
 
 
    Query 9 (and friends) - Grouping and Aggregation
    
+
 
    Also like SQL, SQL++ supports grouped aggregation.
-For each brewery that offers more than 30 beers,
-the following group-by or aggregate query reports the number of beers that it offers.
    
+For each brewery that offers more than 30 beers, the following group-by or aggregate query reports the number of
+beers that it offers.
    
 
 
    SELECT br.brewery_id, COUNT(*) AS num_beers
 FROM beers br
 GROUP BY br.brewery_id
-HAVING num_beers > 30
-ORDER BY num_beers DESC;
+HAVING COUNT(*) > 30
+ORDER BY COUNT(*) DESC;
 
    
 
    
 
-
    The FROM clause incrementally binds the variable br to beers,
-and the GROUP BY clause groups
-the beers by their associated brewery id.
+
    The FROM clause incrementally binds the variable br to beers, and the GROUP
+BY clause groups the beers by their associated brewery id.
 Unlike SQL, where data is tabular (flat), the data model underlying SQL++ allows for nesting.
-Thus, due to the GROUP BY clause, the SELECT clause in this query sees a sequence of br groups,
-with each such group having an associated brewery_id variable value
-(that is the producing brewery’s id).
-In the context of the SELECT clause, brewery_id is bound to
-the brewery’s id and br
-is now re-bound (due to grouping) to the set of beers issued by that brewery.
-The SELECT clause yields a result object containing the brewery’s id and the count of the items
-in the associated beer set.
+Thus, due to the GROUP BY clause, the SELECT clause in this query sees a sequence of br groups, with each such
+group having an associated brewery_id variable value (that is the producing
+brewery's id).
+In the context of the SELECT clause, brewery_id is bound to the brewery's id
+and br is now re-bound (due to grouping) to the set of beers issued by that
+brewery.
+The SELECT clause yields a result object containing the brewery's id and the count of the items in the associated
+beer set.
 The query result will contain one such object per brewery id.
    
 
 
    Below is the expected result for this query over the sample data:
    
@@ -1258,14 +1285,14 @@ 
    Query 9
 (hashing) be used in processing a particular SQL++ query.
    
 
 
    The following query is nearly identical to the previous one, but adds a hash-based aggregation hint
-(though that’s not necessarily the more efficient approach here):
    
+(though that's not necessarily the more efficient approach here):
    
 
 
    SELECT br.brewery_id, COUNT(*) AS num_beers
 FROM beers br
 /*+ hash */
 GROUP BY br.brewery_id
-HAVING num_beers > 30
-ORDER BY num_beers DESC;
+HAVING COUNT(*) > 30
+ORDER BY COUNT(*) DESC;
 
    
 
    
 
@@ -1321,15 +1348,16 @@ 
    Query 9

    Query 10 - Grouping and Limits

    +

    In some use cases it is not necessary to compute the entire answer to a query. In some cases, just having the first N or top N results are sufficient. This is expressible in SQL++ using the LIMIT clause combined with the ORDER BY clause. -(You may have noticed that we have used the LIMIT clause all along to keep the -result set sizes in this document manageable.)

    +(You may have noticed that we have used the LIMIT clause all along to keep the result set sizes in this document +manageable.)

    The following SQL++ query returns the top three breweries based on their numbers of offered beers. -It also illustrates the use of multiple aggregate functions to compute various alcohol content statistics -for their beers:

    +It also illustrates the use of multiple aggregate functions to compute various alcohol content statistics for their +beers:

    SELECT bw.name,
    COUNT(*) AS num_beers,
@@ -1373,15 +1401,15 @@ 
    Query 10 - Grouping and

    Everything Must Change

    +

    So far you have been walking through the SQL++ query capabilities of Analytics. What really makes Analytics interesting, however, is that it brings this query power to bear on your nearly-current -Couchbase Server -data, enabling you to harness the power of parallelism in Analytics to analyze what’s going on with your data -"up front" in essentially -real time, without perturbing your Couchbase Server applications’ performance (or the resulting end user experience). -Before closing this tutorial, let’s take a very quick look at that aspect of Analytics.

    +Couchbase Server data, enabling you to harness the power of parallelism in Analytics to analyze what's going on with +your data "up front" in essentially real time, without perturbing your Couchbase Server applications' performance (or +the resulting end user experience). +Before closing this tutorial, let's take a very quick look at that aspect of Analytics.

    -

    To start, the following SQL++ query lists all of the Kona Brewery’s current beer offerings:

    +

    To start, the following SQL++ query lists all of the Kona Brewery's current beer offerings:

    SELECT meta(b).id, b as beer FROM beers b
 WHERE b.brewery_id = "kona_brewing"
@@ -1508,9 +1536,9 @@ 
    Everything Must C
    -

    To illustrate Analytics in action, suppose that Kona’s marketing department determines that a light beer is needed. -You can use your favorite Couchbase Server interface to modify the server’s beer-sample content accordingly, -for example, a N1QL insert query:

    +

    To illustrate Analytics in action, suppose that Kona's marketing department determines that a light beer is needed. +You can use your favorite Couchbase Server interface to modify the server's beer-sample content accordingly, for +example, a N1QL insert query:

    INSERT INTO `beer-sample` ( KEY, VALUE )
   VALUES
@@ -1522,8 +1550,8 @@ 
    Everything Must C
    -

    Analytics will shadow this change, updating the beers shadow dataset as a result. -Go ahead and rerun the Analytics SQL++ query that lists the Kona Brewery’s beer offerings:

    +

    Analytics will shadow this change, updating the beers dataset as a result. +Go ahead and rerun the Analytics SQL++ query that lists the Kona Brewery's beer offerings:

    SELECT meta(b).id, b as beer FROM beers b
 WHERE b.brewery_id = "kona_brewing"
@@ -1666,21 +1694,20 @@ 
    Everything Must C
    -

    To further illustrate Analytics in action, suppose that Kona’s CEO determines that a light beer is in fact NOT needed. -You can use your favorite N1QL interface again to modify the Couchbase Server’s beer-sample content accordingly, -that is:

    +

    To further illustrate Analytics in action, suppose that Kona's CEO determines that a light beer is in fact NOT needed. +You can use your favorite N1QL interface again to modify the Couchbase Server's beer-sample content accordingly, that + is:

    DELETE FROM `beer-sample` b USE KEYS "kona_brewing-skimboard_light_ale";
    -

    Finally, if you run the SQL++ Kona beer list query on Analytics once again, you will find that the Kona CEO’s -wishes have been -shadowed in Analytics as well.

    +

    Finally, if you run the SQL++ Kona beer list query on Analytics once again, you will find that the Kona CEO's +wishes have been shadowed in Analytics as well.

    And of Course, Indexing

    -

    Last but not least, if you’re a database fan who has come this far, you may be wondering about indexing. +

    Last but not least, if you're a database fan who has come this far, you may be wondering about indexing. Indeed, in addition to the use of efficient parallel execution strategies for large analytical queries, Couchbase Analytics provides support for the use of indexes to speed the execution of smaller queries. For example, consider the following query:

    @@ -1699,27 +1726,27 @@

    And of Course, Indexing

    -

    Note that index creation requires temporary suspension of the shadowing process, although queries -on the accumulated data can continue to be run while shadowing is on hold and the index is being built. -Note also that a CREATE INDEX statement needs to specify the type as well as the name of the field to be -indexed; the above brewerIndex will be used to speed queries that have string-based brewery_id predicates.

    +

    Note that index creation requires temporary suspension of the shadowing process, although queries on the accumulated +data can continue to be run while shadowing is on hold and the index is being built. +Note also that a CREATE INDEX statement needs to specify the type as well as the name of the field to be indexed; the +above brewerIndex will be used to speed queries that have string-based brewery_id predicates.

    Further Help

    -

    That’s it! You are now armed and dangerous with respect to semistructured data management using Analytics via SQL++. + +

    That's it! You are now armed and dangerous with respect to semistructured data management using Analytics via SQL++. For more information, see SQL++ Language Reference, or consult the complete list of built-in functions in the Function Reference.

    -

    Couchbase Analytics lets you bring a powerful new NoSQL parallel query engine to bear on your data, -using the latest state of the art parallel query processing techniques under the hood. We hope you find it useful -in exploring and analyzing your Couchbase Server data - without having to worry about end user performance impact -or doing ETL grunt work to make your analyses possible. -(Note that the DP2 demo set up does not effectively demonstrate the performance aspect of Analytics.)

    +

    Couchbase Analytics lets you bring a powerful new NoSQL parallel query engine to bear on your data, using the latest +state of the art parallel query processing techniques under the hood. +We hope you find it useful in exploring and analyzing your Couchbase Server data - without having to worry about end +user performance impact or doing ETL grunt work to make your analyses possible. +(Note that the DP demo set up does not effectively demonstrate the performance aspect of Analytics.)

    Use it wisely, and remember: "With great power comes great responsibility…" :-)

    Do let us know how you like it…!

    -
    diff --git a/content/analytics/quick-start.html b/content/analytics/quick-start.html index ad78517097..99779ba434 100644 --- a/content/analytics/quick-start.html +++ b/content/analytics/quick-start.html @@ -3,16 +3,15 @@ - Quick Start
    -

    Quick Start

    +

    Using this section, get Analytics up and running quickly, and learn how to run analytic queries on rich, flexible JSON documents. Try it out and let us know what do you think.

    @@ -32,7 +31,7 @@

    Using Docker

    at http://localhost:8091 with the username Administrator and the password password. Note that you might need "sudo" to run the Docker command, depending on your system configuration.

    -

    Later on, when you’re all done experimenting, you can stop the Docker image by running the following command:

    +

    Later on, when you're all done experimenting, you can stop the Docker image by running the following command:

    docker stop analytics
    @@ -47,7 +46,7 @@

    Installing On-Premi

    After installation you need to Initialize the Cluster.

    -

    What’s Next

    +

    What's Next

    Now you can continue to Running Your First Analytics Query to get familiar with the ways you can run a query or continue directly to the tutorial.

    diff --git a/content/analytics/releasenote.html b/content/analytics/releasenote.html index bc1dc45213..1696ce705a 100644 --- a/content/analytics/releasenote.html +++ b/content/analytics/releasenote.html @@ -3,36 +3,30 @@ - Release Note
    -

    Release Note

    + -

    What’s new?

    +

    What's new?

    The following list covers new features in Couchbase Analytics Developer Preview 4.

    - - - - - - - - - - - +
    IssueDescriptionDetail
    + + + + + @@ -65,7 +59,7 @@

    What’s new?

    IssueDescriptionDetail

    Limitations

    @@ -73,15 +67,11 @@

    Limitations

    The following list covers some of the limitations with Couchbase Analytics Developer Preview 4. Please read this carefully before reporting any new issues.

    - - - - - - - - - +
    IssueDescription
    + + + + diff --git a/content/analytics/rest-analytics.html b/content/analytics/rest-analytics.html index 19ecf8e2a3..072ee39d5d 100644 --- a/content/analytics/rest-analytics.html +++ b/content/analytics/rest-analytics.html @@ -3,16 +3,15 @@ - Analytics REST API
    -

    Analytics REST API

    +

    Analytics provides REST APIs that a client application can use to invoke services using simple HTTP methods. A catalogue of available REST resources and requests is provided below.

    @@ -22,7 +21,7 @@

    Analytics REST API

    For the examples we assume that

      -
    • you’re running a Couchbase node with the Analytics service using the default port on localhost and that
      • +
    • you're running a Couchbase node with the Analytics service using the default port on localhost and that
    • you authenticate as a user with the "Full Administrator" role with the user name "Administrator" and the password "password".
    @@ -148,7 +147,6 @@

    GET /analytics/cluster

    PUT /analytics/cc/configPUT /analytics/node/config -

    Description Modifies a node’s configuration.

    +

    Description Modifies a node's configuration.

    Attention: For the configuration changes to take effect, the Analytics cluster must be restarted using the Restart API. diff --git a/content/analytics/run-query.html b/content/analytics/run-query.html index 0b8e695e50..e89c65a62c 100644 --- a/content/analytics/run-query.html +++ b/content/analytics/run-query.html @@ -3,25 +3,24 @@ - Running query
    -

    Running Queries

    +

    You can run queries using the Analytics Workbench, the command line query tool cbq, or directly through the REST API. For the examples we assume that

      -
    • you’re running a Couchbase node with the Analytics service on localhost (the DNS name in the URL could be different +
    • you're running a Couchbase node with the Analytics service on localhost (the DNS name in the URL could be different in a Docker or a real cluster setup) and that
    • you authenticate as a user with the "Full Administrator" role with the user name Administrator and the password -password.
      • +password (of course you should use different credentials for your installation).

    Using the Analytics Workbench

    @@ -40,6 +39,9 @@

    Using

    +

    The result of this query is an array that contains an single object containing a field "greeting" with the value +"Hello, beer!":

    +
    Note: You must press the Execute @@ -63,7 +65,7 @@

    Using the 
    cbq> SELECT "Hello, beer!" AS greeting;
 {
-    "requestID": "b8f514bf-8081-47ba-8f03-88c918e5a048",
+    "requestID": "dfb78524-0a2b-4dd3-8b99-1554d8dc0bcd",
     "signature": "*",
     "results": [ {
         "greeting" : "Hello, beer!"
@@ -71,15 +73,19 @@ 
    Using the
     ,
     "status": "success",
     "metrics": {
-        "elapsedTime": "9.370248ms",
-        "executionTime": "7.573209ms",
+        "elapsedTime": "21.947426ms",
+        "executionTime": "18.042413ms",
         "resultCount": 1,
-        "resultSize": 34
+        "resultSize": 34,
+        "processedObjects": 0
     }
 }
    +

    Here you can see that the result shown in the Analytics Workbench is embedded in an envelope that contains additional +information about the request like the requestID, the status, and some metrics.

    +

    Using the REST API

    You can use the REST API to run Analytics queries. @@ -96,21 +102,25 @@

    Using the REST APIThe request returns:

    {
-    "requestID": "f981ced1-efc0-4087-bae9-5d4cabbd42cc",
+    "requestID": "f1b8451c-f626-46be-b1f8-e9ffc891550b",
     "signature": "*",
-    "results": [ { "greeting": "Hello, beer!" } ],
+    "results": [ { "greeting": "Hello, beer!" }
+    ]
+    ,
     "status": "success",
     "metrics": {
-        "elapsedTime": "10.161083ms",
-        "executionTime": "8.507581ms",
+        "elapsedTime": "28.292741ms",
+        "executionTime": "24.82841ms",
         "resultCount": 1,
-        "resultSize": 31
+        "resultSize": 31,
+        "processedObjects": 0
     }
 }
    -

    In addition to submitting queries to the Analytics Service you can also use the REST API for administrative purposes. +

    again including the envelope that we've seen in the example above. +In addition to submitting queries to the Analytics Service you can also use the REST API for administrative purposes. E.g. you can retrieve version information for the Analytics Service with

    $ curl http://localhost:8095/analytics/version
diff --git a/content/analytics/troubleshoot.html b/content/analytics/troubleshoot.html
index ee5f2c3cb9..2b7de463dc 100644
--- a/content/analytics/troubleshoot.html
+++ b/content/analytics/troubleshoot.html
@@ -3,15 +3,14 @@
   
     
     
-    Troubleshooting and FAQs
   
   
     
    
     
    
       
-
    Troubleshooting and FAQs
    
+
 
 
    This section lists most common errors and gives general tips.
    
 
@@ -73,7 +72,7 @@ 
    Reporting an Issue
    
 
 
    Sending Feedback
    
-
    We’d love to hear from you! Feel free to post your feedback to the
+
    We'd love to hear from you! Feel free to post your feedback to the
 Couchbase Forum.
    
 
 
    FAQs
    
diff --git a/content/architecture/connectivity-architecture.dita b/content/architecture/connectivity-architecture.dita
index f05fa18cf7..78f457b312 100644
--- a/content/architecture/connectivity-architecture.dita
+++ b/content/architecture/connectivity-architecture.dita
@@ -7,7 +7,7 @@
   
    Client to Cluster Communication Client
    applications communicate with Couchbase Server through a set of access points tuned for the data
    access category such as CRUD operations, N1QL queries, and so on. Each access point supports
-   clear text and encrypted communication ports. 
     There are four main types of access points that
+   clear text and encrypted communication ports. 
     There are five main types of access points that
     drive the majority of client to server communications.

    IssueDescription
    Communication ports diff --git a/content/architecture/core-data-access-bucket-expiration.dita b/content/architecture/core-data-access-bucket-expiration.dita new file mode 100644 index 0000000000..deda39d042 --- /dev/null +++ b/content/architecture/core-data-access-bucket-expiration.dita @@ -0,0 +1,294 @@ + + + + + + + + Bucket TTL + + + + Bucket Time To Live (TTL) imposes a maximum lifespan on documents within a bucket. + + + + +
    + + + Overview + + +

    + Bucket TTL is a non-negative integer-value, specified per bucket, that determines the maximum expiration times + of individual documents: if Bucket TTL is enabled, each newly created + document lives for the number of seconds specified by Bucket TTL, following the document's creation. When its + expiration time is reached, the document + is marked for deletion. Deleted documents and their contents are no longer available: access-attempts are treated + as if the documents never existed. +

    + +

    + The maximum value for Bucket TTL is MAX32INT seconds (2147483648, or 68.096 years). The default value is 0, which + indicates that Bucket TTL is disabled. If Bucket TTL is changed from the default, it is thereby enabled. +

    + +

    + Whenever Bucket TTL is modified, documents that existed prior to the modification: +

    + +
      +
    • + Remain subject to the previous Bucket TTL, if Bucket TTL was previously enabled. + +

      + +

      + +
      • + +
    • + Remain subject to no Bucket TTL, if Bucket TTL was not previously enabled. + +

      + +

      + +
      • + +
    + +

    + Documents created or modified following the modification of Bucket TTL are subject to the modified Bucket TTL. +

    + +

    + Bucket TTL can be established on Couchbase and Ephemeral buckets. It cannot be established on + Memcached buckets. Note that Bucket TTL is only available in the Enterprise Edition of Couchbase Server. +

    + + +
    + +
    + + + Expiration: Bucket versus Document + + +

    + As described in the Expiration Overview section of + Core Operations, + an expiration time can be specified per document, by means of Couchbase SDK APIs: this is referred + to as the document’s TTL. If a new document’s TTL is specified to be: + +

    + +
      +
    • + Above that of Bucket TTL, the document's TTL is reduced to the value of Bucket TTL. + +

      + +

      + +
      • + +
    • + Below that of Bucket TTL, the document's TTL is left unchanged. + +

      + +

      + +
      • + +
    • + 0, the document's TTL is reset to the value of Bucket TTL. + +

      + +

      + +
      • + +
    + +
    + +
    + + + Bucket TTL Use Cases + + +

    + Bucket TTL is intended for use with buckets containing data that either must or can be permitted to expire after + a certain period of time. +

    + +

    + Bucket TTL must not be used with buckets whose data is required not to expire: for example, the buckets + that support Couchbase Eventing and Couchbase Mobile. If Bucket TTL is used with such buckets, system failures + may result. +

    + +
    + +
    + + + Post-Expiration Purging + + +

    + When its expiration time is reached, a document is marked for deletion, and is deleted on the next + access-attempt. + As described in + Bucket Disk Storage, + for each document that is deleted, a tombstone, which includes key and metadata, continues to be maintained by + Couchbase Server for some period of time, as a record: this applies to all instances of document-deletion, + including those achieved by specifying Bucket TTL. To ensure that no trace of deleted documents remains, + tombstones are + removed through a Metadata Purge. + This is an automatic, non-disruptive background-process, which can be scheduled if necessary by means of + Couchbase Web Console; as described in + Configuring Auto-Compaction. + Note that the console allows the intervals between purges to be established as appropriately low, so + that tombstones are removed promptly. +

    + +
    + +
    + + + Bucket-Expiration and XDCR + + +

    + When Cross Data-Center Replication occurs, the Bucket TTL setting does not get propagated from the source bucket + to the target. However, documents within the source bucket that have individual expiration times, including ones + derived from the source Bucket TTL setting, are replicated to the target along with their individual expiration times. +

    + +

    + Buckets in the target cluster can have their own Bucket TTL settings, which may differ from those + of the buckets in the source. If, on the target cluster, the TTL of a replicated document is: +

    + +
      +
    • + Above that of Bucket TTL, the document's TTL is reduced to the value of Bucket TTL. + +

      + +

      +
      • + +
    • + Below that of Bucket TTL, the document's TTL is left unchanged. + +

      + +

      + +
      • + +
    • + 0, the document's TTL is reset to the value of Bucket TTL. + +

      +

      +
      • +
    + +

    + Note that in cases where the TTL of the replicated document is above that of Bucket TTL or is 0, if bi-directional + XDCR has been set up, the TTL of the document at the source is + eventually also either reduced or reset to the value of the target's Bucket TTL. +

    + +

    + Note also that when a document is replicated by XDCR, its expiration time is communicated to the target as absolute. + This requires that the system clocks of the respective clusters be fully synchronized; otherwise, inconsistent + behavior may result (as, for example, in the case where a document arrives at the target-cluster with an absolute + expiration time that is earlier than the current time acknowledged by the target-cluster). See + Clock Sync with NTP. +

    + +
    + +
    + + + Setting Bucket-Expiration + + +

    + An expiration time can be set for a bucket by means of any of the following: +

    + +
      + +
    • + The APIs of the Couchbase SDK. See + Core Operations. +

      +

      + +
      • + +
    • + The UI provided by Couchbase Web Console. See + Create a Bucket. +

      +

      + +
      • + +
    • + The Couchbase CLI. See + bucket-create. + +

      +

      + +
      • + +
    • + The Couchbase REST API. See + Creating and Editing Buckets. +

      +

      + +
      • + +
    + +

    + For information on roles that allow modification of bucket-settings, see + Authorization. +

    + +
    + +
    + + + Auditing + + +

    + If auditing is switched on, changes to each bucket's expiration time are + recorded, and can be subsequently viewed. See + Auditing. +

    + +
    + + + +     diff --git a/content/architecture/core-data-access-buckets.dita b/content/architecture/core-data-access-buckets.dita index d389e91167..516d49bf4b 100644 --- a/content/architecture/core-data-access-buckets.dita +++ b/content/architecture/core-data-access-buckets.dita @@ -57,7 +57,7 @@
  • - Full: All data — including keys, key-values, and meta-data — is removed. Generally, + Full: All data — including keys, key-values, and metadata — is removed. Generally, this favors memory at the expense of performance.
    • @@ -322,6 +322,20 @@ Yes Yes + + + Bucket TTL + No + Yes + Yes + + + + Data Compression + No + Yes + Yes + @@ -427,7 +441,7 @@ - Auto-failover + Auto-failover and auto-reprovision @@ -438,8 +452,8 @@ - Auto-failover starts as soon as a node is inaccessible. - Auto-failover can happen multiple times, for multiple nodes. When a failed node becomes + Auto-reprovision starts as soon as a node is inaccessible. + Auto-reprovision can happen multiple times, for multiple nodes. When a failed node becomes accessible again, no delta-node recovery is required, since no data resides on disk. diff --git a/content/architecture/core-data-access-compression.dita b/content/architecture/core-data-access-compression.dita new file mode 100644 index 0000000000..cb697b48e0 --- /dev/null +++ b/content/architecture/core-data-access-compression.dita @@ -0,0 +1,273 @@ + + + + + + + + Data Compression + + + + Couchbase Server supports data compression in its communications + with internal and external clients, and in its internal handling of documents. + + + + +
    + + + Overview + + +

    + The compression of data allows RAM and disk-space to be used with increased efficiency. It may + also reduce consumption of network bandwidth. Higher consumption of CPU resources may result. +

    + +

    + Compression, provided by the open-source library + Snappy, + is applied to documents based both on the client's capabilities and on the compression mode established + by the user for the given bucket. +

    + +

    + Compression is available only in Couchbase Enterprise Edition, and can be applied only to Couchbase and + Ephemeral buckets. +

    + +
    + +
    + + + Where Compression is Used + + +

    + +

    + +

    + Clients based on the Couchbase SDK (1), nodes within a cluster that participate + in intra-cluster replication (4), internal Couchbase Services (5), + external DCP clients (6), and remote clusters that participate in Cross Data-Center + Replication (7) communicate their ability to send and receive compressed + documents by using the HELO command, with a flag that confirms support of the Snappy + Compression data type. +

    + +

    + Couchbase Server may (depending on the mode of the bucket) store documents in compressed form in + memory (2). The server always compresses + documents when storing them on disk (3). +

    + +
    + +
    + + + Compression Modes + + +

    + Each bucket + is configured to support one of three modes. After a client has communicated + its ability to send and receive compressed data, the server's running of compression and decompression routines + depends on the + mode supported by the specified bucket. The modes are as follows: +

    + +

    + +

    + +
      + +
    • + + Off: Provides the + behavior of Couchbase Server pre-5.5. On receipt of a compressed document, + Couchbase Server decompresses the document when storing in memory; and recompresses it + when storing on disk. Couchbase Server + sends the document in uncompressed form. + This mode is assigned by default to buckets upgraded from a previous + version of Couchbase Server. + The mode is recommended for use where clients cannot benefit from compression, and + where neither memory-resources nor network-bandwidth will be negatively impacted by the size and + quantity of documents to be handled. Note also that this is the only mode under which Memcached buckets + operate. + + +
      • + +
    • + Passive: On receipt of a compressed document, + Couchbase Server stores it in compressed form both in memory + and on disk. Couchbase Server + sends the document back to the client in compressed form if this is requested by the client; otherwise, + it sends the document back in uncompressed form. + On receipt of an uncompressed document, Couchbase Server + stores it in memory in uncompressed form, and stores it on disk in compressed form. + It returns the document to the client in uncompressed form. + This mode is assigned by default to new buckets, in + Couchbase Server 5.5 and beyond. It supports clients that themselves handle compresion, and + additionally allows Couchbase Server to + to limit its use of memory-resources and network-bandwidth. At the same time, it does not force + Couchbase Server to use CPU + resources for the compression and decompression of documents that clients do not + themselves require in compressed form. + + +
      • + +
    • + Active: Couchbase Server actively compresses documents for storage in memory + and on disk, even if the documents are received in uncompressed form. + Documents are decompressed before being sent back to those clients that do not + support the receiving of compressed data. Documents are sent + in compressed form to clients that do support the receiving of compressed data, even if those + clients originally sent the + documents to the server in uncompressed form. + This mode allows the server to practice the maximum conservation of memory-resources + and network-bandwith. Consequently, more documents can be held in memory simultaneously, and thereby + accessed with improved, overall efficiency (since the processing-time required for compression and + decompression is significantly less than that required for fetching data from disk). Nevertheless, in some circumstances, + clients that do not themselves require the compression and decompression of documents may be negatively affected by + the compression and decompression performed by the server on documents resident in memory. +
      • + +
    + +
    + + +
    + + + Switching Between Compression Modes + + +

    + Buckets can be switched between modes. The following behavior-changes + should be noted: +

    + +
      + +
    • + When a bucket formerly in Passive mode has been switched to Off mode, + any compressed document that Couchbase Server receives for that bucket + is stored in memory in uncompressed form. If the server + needs to send from the bucket a document that is currently compressed, the server decompresses + the document before sending: however, to preserve + memory-efficiency, the + document remains compressed in memory. + +

      + +

      + +
      • + +
    • + When a bucket has been switched from Passive to Active mode, it + periodically runs a task that compresses uncompressed documents. + +

      + +

      + +
      • + +
    • + When a bucket formerly in Active mode is switched to Off mode, + this disables + the task that was periodically run to compress uncompressed documents. + Compressed documents can continue to be received for the bucket, and are + decompressed for storage in memory. Compressed documents within the bucket are + decompressed before sending: however, to ensure continued + memory-efficiency, the + document remains compressed in memory. + +

      + +

      + +
      • + +
    + +
    + + + +
    + + + Enabling Compression + + +

    + Couchbase Server allows authorized users to configure + compression by means of the following. +

    + +
      + +
    • +

      + The Couchbase Web Console, + the Couchbase CLI, and the Couchbase REST API, which allow buckets each to be assigned a compression mode; and which allow + Cross Data-Center Replication (XDCR) to handle compressed documents. +

      + +

      + For information on using the Couchbase Web Console to create buckets, see + Create a Bucket; and + for information on using it to set up XDCR, see + Managing XDCR +

      + +

      + +

      +
      • + + +
    • +

      + The cbbackup, cbrestore, and + associated tools, which allow + compressed documents to be requested in the backup stream. +

      +

      + +

      +
      • + +
    • +

      + The Couchbase SDK, which can elect to send and receive compressed documents. +

      +

      + +

      +
      • + +
    + +

    + For information on roles that allow modification of bucket-settings, see + Authorization. +

    + +
    + + + +     diff --git a/content/architecture/db-engine-architecture.dita b/content/architecture/db-engine-architecture.dita index 48c0ea1443..b64ccc5968 100644 --- a/content/architecture/db-engine-architecture.dita +++ b/content/architecture/db-engine-architecture.dita @@ -244,21 +244,8 @@

    You can configure the bucket priority settings at the bucket level and set the value to be either high or low. Bucket priority settings determine whether I/O tasks for a bucket must be queued in the low or high priority task queues. Threads in the global pool poll the high priority task queues more often than the low priority task queues. When a bucket has a high priority, its I/O tasks are picked up at a higher frequency and thus, processed faster than the I/O tasks belonging to a low priority bucket.

    You can configure the bucket I/O priority settings during initial setup and change the settings later, if needed. However, changing a bucket I/O priority after the initial setup - results in a restart of the bucket and the client connections are reset. - Create bucket settings - -

    -

    The previous versions of Couchbase Server, version 3.0 or earlier, required the I/O thread - allocation per bucket to be configured manually. However, when you upgrade from a 2.x version to - a 3.x or higher version, Couchbase Server converts an existing thread value to either a high or - low priority based on the following criteria:

      -
    • Buckets allocated six to eight (6-8) threads in Couchbase Server 2.x are marked high - priority in bucket setting after the upgrade to 3.x or later.
      • -
    • Buckets allocated three to five (3-5) threads in Couchbase Server 2.x are marked low - priority in bucket settings after the upgrade to 3.x or later.
      • -

    + results in a restart of the bucket and the client connections are reset. +

    Monitoring Scheduler diff --git a/content/architecture/global-secondary-indexes.dita b/content/architecture/global-secondary-indexes.dita index 9f06a966e9..f4ec8b2fc5 100644 --- a/content/architecture/global-secondary-indexes.dita +++ b/content/architecture/global-secondary-indexes.dita @@ -69,9 +69,17 @@
    Index Storage Options

    Index service can be configured to utilize standard or the memory-optimized storage option.

    Memory-Optimized GSI: Indexes are created for achieving low latency query execution. When memory-optimized storage option is selected, all indexes on the cluster utilize fast in-memory index processing for efficient index maintenance and index scans. Memory optimized indexes uses the skiplist structure as opposed to B-tree indexes. Skiplist structure optimize memory consumption. They also enhance concurrent processing of index updates and scans with a lock-free processing arrangements of the index in memory. Memory optimized indexes provide the most optimized index for processing high velocity mutations and high frequency scans.

    -

    When standard storage option is selected, the supervisor process uses the ForestDB storage engine to persist changes to the individual indexes to disk. Each index gets a dedicated file. With standard storage mode, writes to the storage can either be done using an append-only write or using a write with circular reuse depending on the write mode selected for global secondary indexes.

    -

    Both write modes in the standard storage mode for GSI requires compaction to clean up the orphaned space. Writes with circular reuse require compaction much less frequently compared to the append only writes. For more information about compaction and differences between ForestDB and Couchstore, and write modes in ForestDB, see Storage Architecture.

    +

    When standard storage option is selected, the supervisor process uses the storage engine to persist changes to the individual indexes to disk. Each index gets a dedicated file. With standard storage mode, writes to the storage can either be done using an append-only write or using a write with circular reuse depending on the write mode selected for global secondary indexes.

    +

    Both write modes in the standard storage mode for GSI requires compaction to clean up the orphaned space. + Writes with circular reuse require compaction much less frequently compared to the append only writes.

    + +

    + See + Storage Architecture for more information. +

    + +
    GSI and High Availability

    Unlike the database engine or the views engine, Global Secondary Indexes do not provide automatic built-in replicas. As an administrator, you can manually create replicas with GSIs by using the index service to create identical indexes on separate nodes. To create redundant copies of an index, you can create the same index definition with different index names on different nodes that are running the index service.

    diff --git a/content/architecture/gsi-versus-views.dita b/content/architecture/gsi-versus-views.dita index acb135c0ff..4fe14d081f 100644 --- a/content/architecture/gsi-versus-views.dita +++ b/content/architecture/gsi-versus-views.dita @@ -59,8 +59,8 @@ Views depend on the file system cache and can be impacted by fluctuations in the underlying operating system. - Storage engine For more information on the differences between Couchstore and ForestDB, see "Couchstore versus ForestDB" under Storage Architecture. - GSIs use ForestDB for storage. + Storage engine + Enterprise Edition GSIs use Plasma for storage. Community Edition GSIs use ForestDB. Views use Couchstore for storage. diff --git a/content/architecture/images/compression-overview-diagram.png b/content/architecture/images/compression-overview-diagram.png new file mode 100644 index 0000000000..9c072d966c Binary files /dev/null and b/content/architecture/images/compression-overview-diagram.png differ diff --git a/content/architecture/querying-data-with-views.dita b/content/architecture/querying-data-with-views.dita index 9e470aab53..20fb8ea6b9 100644 --- a/content/architecture/querying-data-with-views.dita +++ b/content/architecture/querying-data-with-views.dita @@ -16,7 +16,7 @@

    -

    When a view query is issued, one of the data service nodes receives it and becomes the coordinator node that is responsible to execute the query. Given that the partitioning of the view is based on document keys and views are queries based on view keys, the query gets scattered to all nodes by the coordinator. In the scatter phase, each node running the data service is asked to execute the query on their local view partition. The results from each node running the data service is then gathered by the coordinator node. The coordinator node compiles the final results, applying the required parameters again to limit, skip, and sort the output for final delivery to the client.

    +

    When a view query is issued, one of the data service nodes receives it and becomes the coordinator node that is responsible for executing the query. Given that the partitioning of the view is based on document keys, and views are queries based on view keys, the query gets scattered to all nodes by the coordinator. In the scatter phase, each node running the data service is asked to execute the query on its local view partition. The results from each node running the data service are then gathered by the coordinator node. The coordinator node compiles the final results, applying the required parameters again to limit, skip, and sort the output for final delivery to the client.

    Query Consistency with Views

    Mutations that arrive at a bucket do not block and wait for view indexing for the mutation to complete. View updates are triggered by the view queries or the automatic view updater. That is, views are eventually consistent with ongoing mutations.

    When querying views, as a developer, you may require varying consistency levels. Consistency levels for any given query can be configured using the staleness parameter for the query. The following consistency levels can be specified for the staleness parameter:

      diff --git a/content/architecture/storage-architecture.dita b/content/architecture/storage-architecture.dita index 1f7ed4dcc7..af02a8fbc8 100644 --- a/content/architecture/storage-architecture.dita +++ b/content/architecture/storage-architecture.dita @@ -7,24 +7,17 @@

      As an administrator, you can independently control data and index storage paths within the file system on a per node basis. This ensures data and index storage can utilize separate I/O subsystems to enable independent tuning and isolation. There are multiple storage engines in use in Couchbase Server:

      • Data Service, MapReduce Views, Spatial Views, and Couchstore

        For core data operations, MapReduce views, and spatial views, Couchbase Server uses Couchstore. Each vBucket is represented as a separate Couchstore file in the file system. Couchstore uses a B+tree structure to quickly access items through their keys. For efficient writes, Couchstore uses an append-only write model for each file for efficient and safe writes.

        • -
      • Index Service, Search Service, and ForestDB +
      • Index Service, Search Service, Plasma, and ForestDB

        For indexing with GSI in the Index service and full-text index in the search service, - Couchbase Server uses ForestDB. With ForestDB, each index is represented as a separate - ForestDB file in the file system. Unlike Couchstore, ForestDB uses a B+trie structure to - quickly access item through its index key. B+trie provides a more efficient tree - structure compared to B+trees and ensures a shallower tree hierarchy to better scale - large item counts and very large index keys. ForestDB offers multiple options for its - writes. ForestDB can be configured to use an append-only write model for each file for - efficient writes which also requires regular compaction for cleanup. ForestDB can also - be configured to use "circular reuse" which allows incoming writes to reuse the existing - orphaned space within the file instead of simply just append only writes. In the - "circular reuse" mode, compaction is still needed but with much less frequency - (typically once a week). For more information on ForestDB and B+trie, see https://github.com/couchbase/forestdb.

        • + Couchbase Server Enterprise Edition uses Plasma; and Couchbase Server Community Edition uses ForestDB. + For information on Plasma, see + Global Secondary Index Storage. +

      Couchstore Versus ForestDB -

      Couchbase Server uses multiple storage engines to optimize specific I/O patterns required by the services. Couchstore is used for storage under data service for both database engine and for view engine. ForestDB is used by the index service for storage of global secondary indexes.

      +

      Couchbase Server uses multiple storage engines to optimize specific I/O patterns required by the services. + Couchstore is used for storage under data service for both database engine and for the view engine. + On Couchbase Server Community Edition, the index service uses ForestDB for the storage of global secondary indexes.

      There are a few similarities between Couchstore and ForestDB.

      • Both come with an append-only write approach. Additionally, ForestDB supports the circular reuse write approach.
      • Both storage engines perform compression using the SNAPPY library when persisting.
        • diff --git a/content/backup-restore/cbbackupmgr-archivelayout.dita b/content/backup-restore/cbbackupmgr-archivelayout.dita index 9c4f7cf4c8..7f00cf118b 100644 --- a/content/backup-restore/cbbackupmgr-archivelayout.dita +++ b/content/backup-restore/cbbackupmgr-archivelayout.dita @@ -6,7 +6,7 @@ The backup archive is used to store all backup data and this section describes the backup archive layout specification. -

        The top level directory of the archive is called the archive backup. The backup archive +

        The top level directory of the archive is called the backup archive. The backup archive contains one or more backup repositories and a logs directory. The logs directory contains logging information for all cbbackupmgr commands run on the backup archive. Each backup repository contains backups for a specific cluster and each backup is done on a cluster diff --git a/content/backup-restore/cbbackupmgr-backup.dita b/content/backup-restore/cbbackupmgr-backup.dita index 43ce0e4f2a..4bb0880b4d 100644 --- a/content/backup-restore/cbbackupmgr-backup.dita +++ b/content/backup-restore/cbbackupmgr-backup.dita @@ -11,7 +11,11 @@

      -->
      Synopsis -

      cbbackupmgr backup [--archive <archive_dir>] [--repo <repo_name>] [--host <url>] [--username <username>] [--password <password>] [--resume] [--purge] [--thread <num>] [--no-progress-bar]

      +

      cbbackupmgr backup [--archive <archive_dir>] [--repo <repo_name>] + [--cluster <url>] [--username <username>] [--password <password>] + [--resume] [--purge] [--thread <num>] [--cacert <file>] [--no-ssl-verify] + [--value-compression <type>] [--no-progress-bar] [--skip-last-compaction]

      +
      Description @@ -24,56 +28,145 @@ Options

      Below are a list of required and optional parameters for the backup command.

      Required -

      --archive <archive_dir>

      - The location of the backup archive directory. -

      --repo <repo_name>

      - The name of the backup repository to backup data into. -

      --host <hostname>

      - The hostname of one of the nodes in the cluster to back up. See the Host Formats - section below for hostname specification details. -

      --username <username>

      - The username for cluster authentication. -

      --password <password>

      - The password for cluster authentication. - +

      +

      + +
      -a,--archive <archive_dir>
      +
      +

      The location of the backup archive directory.

      +
      +       + +
      -r,--repo <repo_name>
      +
      +

      The name of the backup repository to backup data + into\&.

      +
      +       + +
      -c,--cluster <hostname>
      +
      +

      The hostname of one of the nodes in the cluster to back + up. See the Host Formats section below for hostname specification details.

      +
      +       + +
      -u,--username <username>
      +
      +

      The username for cluster authentication. The user must + have the appropriate privileges to take a backup.

      +
      +       + +
      -p,--password <password>
      +
      +

      The password for cluster authentication. The user must + have the appropriate privileges to take a backup. If not password is supplied to + this option then you will be prompted to enter your password.

      +
      +       +
      +

      Optional -

      --resume

      - If the previous backup did not complete successfully it can be resumed from where it - left off by specifying this flag. Note that the resume and purge flags may not be - specified at the same time. -

      --purge

      - If the previous backup did not complete successfully the partial backup will be removed - and restarted from the point of the previous successful backup by specifying this flag. - Note that the purge and resume flags may not be specified at the same time. -

      --threads <num>

      - Specifies the number of concurrent clients to use when taking a backup. Fewer clients - means backups will take longer, but there will be less cluster resources used to complete - the backup. More clients means faster backups, but at the cost of more cluster resource - usage. This parameter defaults to 1 if it is not specified and it is recommended that this - parameter is not set to be higher than the number of CPUs on the machine where the backup - is taking place. -

      --no-progress-bar

      - By default, a progress bar is printed to stdout so that the user can see how long the - backup is expected to take, the amount of data that is being transferred per second, and - the amount of data that has been backed up. Specifying this flag disables the progress bar - and is useful when running automated jobs. -
      +
    +
    +
    + +
    --resume
    +
    +

    If the previous backup did not complete successfully it can + be resumed from where it left off by specifying this flag. Note that the resume and + purge flags may not be specified at the same time.

    +
    +     + +
    --purge
    +
    +

    If the previous backup did not complete successfully the + partial backup will be removed and restarted from the point of the previous successful + backup by specifying this flag. Note that the purge and resume flags may not be + specified at the same time.

    +
    +     + +
    --no-ssl-verify
    +
    +

    Skips the SSL verification phase. Specifying this flag will + allow a connection using SSL encryption, but will not verify the identity of the + server you connect to. You are vulnerable to a man-in-the-middle attack if you use + this flag. Either this flag or the --cacert flag must be specified when using an SSL + encrypted connection.

    +
    +     + +
    --cacert <cert_path>
    +
    +

    Specifies a CA certificate that will be used to verify the + identity of the server being connecting to. Either this flag or the --no-ssl-verify + flag must be specified when using an SSL encrypted connection.

    +
    +     + +
    --value-compression <compression_policy>
    +
    +

    Specifies a compression policy for backed up values. When + Couchbase sends data to the backup client the data stream may contain all compressed + values, all uncompressed values, or a mix of compressed and uncompressed values. To + backup all data in the same form that the backup client recieves it you can specify + "unchanged". If you wish for all values to be uncompressed then you can specify + "uncompressed". This policy will however uncompress any compressed values received + from Couchbase and may increase the backup file size. To compress all values you can + specify "compressed". This will compress any uncompressed values before wiriting them + to disk. The default value for this option is "unchanged".

    +
    +     + +
    -t,--threads <num>
    +
    +

    Specifies the number of concurrent clients to use when + taking a backup. Fewer clients means backups will take longer, but there will be less + cluster resources used to complete the backup. More clients means faster backups, but + at the cost of more cluster resource usage. This parameter defaults to 1 if it is not + specified and it is recommended that this parameter is not set to be higher than the + number of CPUs on the machine where the backup is taking place.

    +
    +     + +
    --no-progress-bar
    +
    +

    By default, a progress bar is printed to stdout so that the + user can see how long the backup is expected to take, the amount of data that is being + transferred per second, and the amount of data that has been backed up. Specifying + this flag disables the progress bar and is useful when running automated jobs.

    +
    +     +
    +
    Host Formats -

    When specifying a host for the backup command the following formats are expected:

    -
      -
    • couchbase://<addr> +

      When specifying a host for the couchbase-cli command the following + formats are expected:

      +
        +
      • +

        + couchbase://<addr> +

        • -
      -
        -
      • <addr>:<port> +
      • +

        + <addr>:<port> +

        • -
      -
        -
      • http://<addr>:<port> +
      • +

        + http://<addr>:<port> +

      -

      It is recommended to use the couchbase://<addr> format for standard installations. The other two formats allow an option to take a port number which is needed for non-default installations where the admin port has been set up on a port other that 8091.

      +

      It is recommended to use the couchbase://<addr> format for + standard installations. The other two formats allow an option to take a port number which is + needed for non-default installations where the admin port has been set up on a port other + that 8091.

    Examples @@ -148,24 +241,105 @@ $ cbbackupmgr backup --archive /data/backups --repo example \
    Environment And Configuration Variables -

    (None)

    +
    + +
    CB_CLUSTER
    +
    +

    Specifies the hostname of the Couchbase cluster to connect + to. If the hostname is supplied as a command line argument then this value is + overridden.

    +
    +     + +
    CB_USERNAME
    +
    +

    Specifies the username for authentication to a Couchbase + cluster. If the username is supplied as a command line argument then this value is + overridden.

    +
    +     + +
    CB_PASSWORD
    +
    +

    Specifies the password for authentication to a Couchbase + cluster. If the password is supplied as a command line argument then this value is + overridden.

    +
    +     + +
    CB_ARCHIVE_PATH
    +
    +

    Specifies the path to the backup archive. If the archive + path is supplied as a command line argument then this value is overridden.

    +
    +     +
    Files -

    bucket-config.json

    - Stores the bucket configuration settings for a bucket. - -

    views.json

    - Stores the view definitions for a bucket. - -

    gsi.json

    - Stores the global secondary index (GSI) definitions for a bucket. - -

    full-text.json

    - Stores the full-text index definitions for a bucket. - -

    shard-*.fdb

    - Stores the key-value data for a bucket bucket. - +
    + +
    + restrictions.json +
    +
    +

    + Keeps a list of restrictions used to ensure data is not restored to the + place. +

    +
    +     + +
    + bucket-config.json +
    +
    +

    + Stores the bucket configuration settings for a bucket. +

    +
    +     + +
    + views.json +
    +
    +

    + Stores the view definitions for a bucket. +

    +
    +     + +
    + gsi.json +
    +
    +

    + Stores the global secondary index (GSI) definitions for a bucket. +

    +
    +     + +
    + full-text.json +
    +
    +

    + Stores the full-text index definitions for a bucket. +

    +
    +     + +
    + shard-*.fdb +
    +
    +

    + Stores the key\-value data for a bucket. +

    +
    +     +
    +
    Synopsis -

    cbbackupmgr restore [--archive <archive_dir>] [--repo <repo_name>] [--start <backup>] [--end <backup>] [--exclude-buckets <bucket_list>] [--include-buckets <bucket_list>] [--disable-bucket-config] [--disable-views] [--disable-gsi-indexes] [--disable-ft-indexes] [--disable-data] [--force-updates][--threads <integer>] [--no-progress-bar]

    +

    cbbackupmgr restore [--archive <archive_dir>] [--repo <repo_name>] [--cluster + <host>] [--username <username>] [--password <password>] [--start <backup>] + [--end <backup>] [--exclude-buckets <bucket_list>] [--include-buckets + <bucket_list>] [--map-buckets <list>] [--disable-bucket-config] [--disable-views] + [--disable-gsi-indexes] [--disable-ft-indexes] [--disable-data] [--force-updates][--threads + <integer>] [--no-progress-bar]

    Description -

    Restores data from the backup archive to a target Couchbase cluster.

    +

    Restores data from the backup archive to a target Couchbase cluster. By default all data, + index definitions, view definitions, full-text index definitions and bucket configuration + data are restored to the cluster unless specified otherwise in the repository's backup + config or though command line parameters when running the restore command. For example, if + you changed bucket configuration settings since your last backup, then restoring a previous + backup will by default overwrite these settings unless you explicitly tell + cbbackupmgr not to restore the bucket settings using the + --disable-bucket-config flag.

    The restore command is capable of restoring a single backup or a range of backups. When restoring a single backup, all data from that backup is restored. If a range of backups is restored, then cbbackupmgr takes into account any failovers that may have occurred in between the time that the backups were originally taken. If a failover did occur in between the backups, and the backup archive contains data that no longer exists in the cluster, then the data that no longer exists is skipped during the restore. If no failovers occurred in between backups then restoring a range of backups restores all data from each backup. If all data must be restored regardless of whether a failover occurred in between the original backups, then data should be restored one backup at a time.

    The restore command is guaranteed to work during rebalances and failovers. If a rebalance is taking place, cbbackupmgr tracks the movement of vBuckets around a Couchbase cluster and ensures that data is restored to the appropriate node. If a failover occurs during the restore then the client will wait 180 seconds for the failed node to be removed from the cluster. If the failed node is not removed in 180 seconds then the restore will fail, but if the failed node is removed before the timeout then data will continue to be restored.

    +

    Note that if you are restoring indexes then it is highly likely that you will need to take + some manual steps in order to properly restore them. This is because by default indexes will + only be built if they are restored to the exact same index node that they were backed up + from. If the index node they were backed up from does not exist then the indexes will be + restored in round-robin fashion among the current indexer nodes. These indexes will be + created, but not built and will required the administrator to manually build them. We do + this because we cannot know the optimal index topology ahead of time. By not building the + indexes the administrator can move each index between nodes and build them when they deem + that the index topology is optimal.

    - Options -

    Below is a list of required and optional parameters for the restore command.

    + Options

    Below is a list of required and optional parameters for the restore + command.

    Required -

    --archive <archive_dir>

    - The directory containing the backup repository to restore data from. - -

    --repo <repo_name>

    - The name of the backup repository to restore data from. - -

    --host <hostname>

    - The hostname of one of the nodes in the cluster to restore data to. -See the Host Formats section below for hostname specification details. - -

    --username <username>

    - The username for cluster authentication. - -

    --password <password>

    - The password for cluster authentication. - +
    + +
    -a,--archive <archive_dir>
    +
    The directory containing the backup repository to restore data from.
    +     + +
    -r,--repo <repo_name>
    +
    The name of the backup repository to restore data from.
    +     + +
    -c,--cluster <hostname>
    +
    The hostname of one of the nodes in the cluster to restore data to. See the Host + Formats section below for hostname specification details.
    +     + +
    -u,--username <username>
    +
    The username for cluster authentication. The user must have the appropriate privileges + to take a backup.
    +     + +
    -p,--password <password>
    +
    The password for cluster authentication. The user must have the appropriate privileges + to take a backup. If not password is supplied to this option then you will be prompted + to enter your password.
    +     +
    Optional -

    --start <backup>

    - The name of the first backup in the backup repository to restore. -If not specified this value will default to the oldest backup in the backup repository. - -

    --end <backup>

    - The name of the last backup in the backup repository to restore. -If not specified this value will default to the most recent backup in the backup repository. - -

    --exclude-buckets <bucket_list>

    - Restores all buckets in a backup that are not specified in <bucket_list>. -This flag cannot be specified at the same time as the --include-buckets flag. -Takes a comma separated list of bucket names. - -

    --include-buckets <bucket_list>

    - Restores only buckets in a backup that are specified in <bucket-list>. -This flag cannot be specified at the same time as the --exclude-buckets flag. -Takes a comma separated list of bucket names. - -

    --disable-views

    - Skips restoring view definitions for all buckets. - -

    --disable-gsi-indexes

    - Skips restoring gsi index definitions for all buckets. - -

    --disable-ft-indexes

    - Skips restoring full-text index definitions for all buckets. - -

    --disable-data

    - Skips restoring all key-value data for all buckets. - -

    --force-updates

    - Forces data in the Couchbase cluster to be overwritten even if the data in the cluster is newer. -By default updates are not forced and all updates use the conflict resolution mechanism in Couchbase to ensure that newer data on the cluster, if any, is not overwritten by older restore data. - -

    --threads <num>

    - Specifies the number of concurrent clients to use when restoring data. -Fewer clients means restores take longer, but there will be less cluster resources used to complete the restore. -More clients means faster restores, but at the cost of more cluster resource usage. -This parameter defaults to 1 if it is not specified and it is recommended that you do not set this parameter to be higher than the number of CPUs on the machine where the restore is taking place. - -

    --no-progress-bar

    - By default, a progress bar is printed to stdout so that the user can see how long the restore is expected to take, the amount of data that is being transferred per second, and the amount of data that has been restored. -Specifying this flag disables the progress bar and is useful when running automated jobs. - -
    +
    + +
    --start <backup>
    +
    The name of the first backup in the backup repository to restore or an index value + which references an incremental backup. Valid index values are any positive integer, + "oldest", and "latest". If a positive integer is used then it should reference the index + of the incremental backup starting from the oldest to the most recent backup. For + example, "1" corresponds to the oldest backup, "2" corresponds to the second oldest + backup, and so on. Specifying "oldest" means that the index of the oldest backup should + be used and specifying "latest" means the index of the most recent backup should be + used. If this flag is not specified then the restore will start with the oldest backup + in the backup repository.
    +     + +
    --end <backup>
    +
    The name of the last backup in the backup repository to restore or an index value + which references an incremental backup. Valid index values are any positive integer, + "oldest", and "latest". If a positive integer is used then it should reference the index + of the incremental backup starting from the oldest to the most recent backup. For + example, "1" corresponds to the oldest backup, "2" corresponds to the second oldest + backup, and so on. Specifying "oldest" means that the index of the oldest backup should + be used and specifying "latest" means the index of the most recent backup should be + used. If this flag is not specified then the restore will end with the most recent + backup in the backup repository.
    +     + +
    --exclude-buckets <bucket_list>
    +
    Restores all buckets in a backup that are not specified in <bucket_list>. This flag + cannot be specified at the same time as the --include-buckets flag. Takes a comma + separated list of bucket names.
    +     + +
    --include-buckets <bucket_list>
    +
    Restores only buckets in a backup that are specified in <bucket-list>. This flag + cannot be specified at the same time as the --exclude-buckets flag. Takes a comma + separated list of bucket names.
    +     + +
    --filter-keys
    +
    Only restore data where the key matches a particular regular expression.
    +     + +
    --filter-values
    +
    Only restore data where the value matches a particular regular expression.
    +     + +
    --enable-bucket-config
    +
    Enables restoring the bucket configuration.
    +     + +
    --disable-views
    +
    Skips restoring view definitions for all buckets.
    +     + +
    --disable-gsi-indexes
    +
    Skips restoring gsi index definitions for all buckets.
    +     + +
    --disable-ft-indexes
    +
    Skips restoring full-text index definitions for all buckets.
    +     + +
    --disable-data
    +
    Skips restoring all key-value data for all buckets.
    +     + +
    --force-updates
    +
    Forces data in the Couchbase cluster to be overwritten even if the data in the cluster + is newer. By default updates are not forced and all updates use Couchbase's conflict + resolution mechanism to ensure that if newer data exists on the cluster that is not + overwritten by older restore data.
    +     + +
    --map-buckets <bucket_mapping>
    +
    Specified when you want to restore a backup to a destination bucket that has a + different name than the bucket that was originally backed up. This parameter takes a + list of mappings since multiple buckets may be restored at the same time. Each bucket + mapping is separated by an "=" and if multiple bucket mappings are specified then they + should be comma separated. If we have two buckets, bucket-1 and bucket-2, and we want to + restore them to renamed-1 and renamed-2 then we would denote the mapping as + "bucket-1=renamed-1,bucket-2=renamed-2".
    +     + +
    --no-ssl-verify
    +
    Skips the SSL verification phase. Specifying this flag will allow a connection using + SSL encryption, but will not verify the identity of the server you connect to. You are + vulnerable to a man-in-the-middle attack if you use this flag. Either this flag or the + --cacert flag must be specified when using an SSL encrypted connection.
    +     + +
    --cacert <cert_path>
    +
    Specifies a CA certificate that will be used to verify the identity of the server + being connecting to. Either this flag or the --no-ssl-verify flag must be specified when + using an SSL encrypted connection.
    +     + +
    -t,--threads <num>
    +
    Specifies the number of concurrent clients to use when restoring data. Fewer clients + means restores will take longer, but there will be less cluster resources used to + complete the restore. More clients means faster restores, but at the cost of more + cluster resource usage. This parameter defaults to 1 if it is not specified and it is + recommended that this parameter is not set to be higher than the number of CPUs on the + machine where the restore is taking place.
    +     + +
    --no-progress-bar
    +
    By default, a progress bar is printed to stdout so that the user can see how long the + restore is expected to take, the amount of data that is being transferred per second, + and the amount of data that has been restored. Specifying this flag disables the + progress bar and is useful when running automated jobs.
    +     +
    Host Formats

    When specifying a host for the restore command the following formats are expected:

    @@ -150,12 +248,26 @@ Size Items Name
    Discussion -

    The restore command works by replaying the data recorded into backup files. During the restore this creates key-value traffic against the cluster that shows up in the form of "set" operations. The restore command replays data from each file in order to guarantee that older backup data does not overwrite newer data. The restore command uses Couchbase's conflict resolution mechanism by default to ensure this behavior. The conflict resolution mechanism can be disable by specifying the --force-updates flag when executing a restore.

    +

    The restore command works by replaying the data recorded in backup files. During a restore + each key-value pair backed up by cbbackupmgr will be sent to the cluster as either a "set" + or "delete" operation. The restore command replays data from each file in order of backup + time to guarantee that older backup data does not overwrite newer backup data. The restore + command uses Couchbase's conflict resolution mechanism by default to ensure this behavior. + The conflict resolution mechanism can be disable by specifying the --force-updates flag when + executing a restore.

    +

    Starting in Couchbase 4.6 each bucket can have different conflict resolution mechanisms. + cbbackupmgr will backup all meta data used for conflict resolution, but since each conflict + resolution mechanism is different cbbackupmgr will prevent restores to a bucket when the + source and destination conflict resolution methods differ. This is done because by default + cbbackupmgr will use the conflict resolution mechanism of the destination bucket to ensure + an older value does not overwrite a newer value. If you want to restore a backup to a bucket + with a different conflict resolution type you can do by using the --force-updates flag. This + is allowed because forcing updates means that cbbackupmgr will skip doing conflict + resolution on the destination bucket.

    Unlike backups, restores cannot be resumed if they fail.

    Environment And Configuration Variables -

    (None)

    @@ -22,6 +23,7 @@ + @@ -29,8 +31,8 @@ - - + + diff --git a/content/cli/alter-role.dita b/content/cli/alter-role.dita index 054bbe6d9c..41467631d0 100644 --- a/content/cli/alter-role.dita +++ b/content/cli/alter-role.dita @@ -108,6 +108,6 @@ - + diff --git a/content/cli/cbbackup-tool.dita b/content/cli/cbbackup-tool.dita index c3b343a1fc..1ab7fbaf85 100644 --- a/content/cli/cbbackup-tool.dita +++ b/content/cli/cbbackup-tool.dita @@ -15,7 +15,12 @@
  • [backup-dir]

    The directory for the backup files to be stored. If an empty directory doesn't exist, it will be created; the parent directory must exist.

    • - + + + The backup directory must only be used by cbbackup and + cbrestore. If you create the backup in a location which may be used by other + processes, like /tmp, the structure of the archive may become corrupted. +

    The following syntax example includes a full backup and two incremental backups for a cluster:

    @@ -23,12 +28,12 @@ cbbackup http://HOST:8091 /backup-42 -m diff -u Administrator -p password cbbackup http://HOST:8091 /backup-42 -m diff -u Administrator -p password - If the backup directory exists, the following backup command will be - automatically incremented. The only way to generate a full backup is to choose an unused + If the backup directory exists, the following backup command will automatically + be incremental. The only way to generate a full backup is to choose an unused directory. If backing up a Couchbase Server Community Edition cluster, then incremental backup - is not available, all backups will be full backups. + is not available — all backups will be full backups. -

    The following syntax example include a full backup, two differential backups, and one +

    The following syntax example includes a full backup, two differential backups, and one accumulative backup for a single node.

    cbbackup couchbase://HOST:8091 /backup-43 -m full --single-node -u Administrator -p password cbbackup couchbase://HOST:8091 /backup-43 -m diff --single-node -u Administrator -p password @@ -38,12 +43,6 @@ cbbackup couchbase://HOST:8091 /backup-43 -m accu --single-node -u Administrator -

    Syntax example of a full backup plus two differentials and one accumulative for a single node.

    - - cbbackup couchbase://HOST:8091 /backup-43 -m full --single-node -u Administrator -p password \ -cbbackup couchbase://HOST:8091 /backup-43 -m diff --single-nodecbbackup \ -couchbase://HOST:8091 /backup-43 -m diff --single-node -u Administrator -p password \ -cbbackup couchbase://HOST:8091 /backup-43 -m accu --single-node -u Administrator -p password] A full backup task is always triggered for a new sink location no matter what backup mode is specified. @@ -250,7 +249,7 @@ cbbackup couchbase://HOST:8091 /backup-43 -m accu --single-node -u Administrator conflict_resolve=1 - By default, disable conflict resolution.

    * This option doesn't work + By default, disable conflict resolution.

    This option doesn't work in Couchbase Server versions 4.0 and 4.1 but will be re-implemented in version 4.1.1 and in subsequent versions.

         @@ -261,7 +260,10 @@ cbbackup couchbase://HOST:8091 /backup-43 -m accu --single-node -u Administrator design_doc_only=0 For value 1, transfer only design documents from a backup file or - cluster. Default: 0. + cluster. Default: 0. +

    Back up only design documents which include view and secondary index definitions from a + cluster or bucket with the option design_doc_only=1. Restore only design + documents with cbrestore -x design_doc_only=1.

     max_retry=10 @@ -308,18 +310,14 @@ cbbackup couchbase://HOST:8091 /backup-43 -m accu --single-node -u Administrator uncompress=0 - For value 1, restore data in uncompressed mode. + For value 1, restore data in uncompressed mode. +

    This option is unsupported. To create backups with compression, you should use + , which is available for + Couchbase Server Enterprise Edition only.

         -
    - - Back up only design documents which includes view and secondary index definitions from a cluster or bucket with the option, - design_doc_only=1. Restore only design documents with - cbrestore design_doc_only=1. - - - + diff --git a/content/cli/cbbackup-wrapper.dita b/content/cli/cbbackup-wrapper.dita index feae749550..b6529d7ac9 100644 --- a/content/cli/cbbackup-wrapper.dita +++ b/content/cli/cbbackup-wrapper.dita @@ -1,9 +1,9 @@ - <cmdname>cbbackupwrapper</cmdname> and <cmdname>cbrestorewrapper</cmdname>Two new tools are - included with Couchbase Server Enterprise Edition - that improve throughput for the database backup process: cbbackupwrapper and cbrestorewrapper. + <cmdname>cbbackupwrapper</cmdname> and <cmdname>cbrestorewrapper</cmdname> + Two tools are included with Couchbase Server Enterprise Edition that improve throughput for the database + backup process: cbbackupwrapper and cbrestorewrapper.

    cbbackupwrapper and cbrestorewrapper were specifically created to address the single worker thread limitation for one backup/restore target (Bucket, @@ -42,36 +42,36 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS -h, --help - Show this help message and exit + Show this help message and exit. -b BUCKET_SOURCE, --bucket-source=BUCKET_SOURCE - Specify the bucket to backup. Defaults to all buckets + Specify the bucket to backup. Defaults to all buckets. --single-node - Use a single server node from the source only + Use a single server node from the source only. -u USERNAME, --username=USERNAME - REST username for source cluster or server node. Default is Administrator + REST username for source cluster or server node. Default is Administrator. -p PASSWORD, --password=PASSWORD - REST password for source cluster or server node + REST password for source cluster or server node. -s, --ssl - Transfer data with SSL enabled + Transfer data with SSL enabled. -v, --verbose - Enable verbose messaging + Enable verbose messaging. --path=PATH Specify the path to cbbackup. Defaults to current - directory + directory. --port=PORT @@ -91,7 +91,7 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS -m MODE, --mode=MODE - Backup mode: full, diff or accu [default:diff] + Backup mode: full, diff or accu [default:diff]. @@ -119,23 +119,23 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS backoff_cap=10 - Max backoff time during rebalance period + Max backoff time during rebalance period. batch_max_bytes=400000 - Transfer this # of bytes per batch + Transfer this # of bytes per batch. batch_max_size=1000 - Transfer this # of documents per batch + Transfer this # of documents per batch. cbb_max_mb=100000 - Split backup file on destination cluster if it exceeds MB + Split backup file on destination cluster if it exceeds MB. data_only=0 - For value 1, only transfer data from a backup file or cluster + For value 1, only transfer data from a backup file or cluster. dcp_consumer_queue_length=1000 @@ -146,7 +146,7 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS design_doc_only=0 For value 1, transfer design documents only from a backup file or - cluster + cluster. flow_control=1 @@ -154,11 +154,11 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS max_retry=10 - Max number of sequential retries if transfer fails + Max number of sequential retries if transfer fails. recv_min_bytes=4096 - Amount of bytes for every TCP/IP call transferred + Amount of bytes for every TCP/IP call transferred. rehash=0 @@ -168,12 +168,12 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS report=5 - Number batches transferred before updating progress bar in console + Number batches transferred before updating progress bar in console. report_full=2000 Number batches transferred before emitting progress information in - console + console. seqno=0 @@ -181,7 +181,10 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS uncompress=0 - For value 1, restore data in uncompressed mode. + For value 1, restore data in uncompressed mode. +

    This option is unsupported. To create backups with compression, you should use + , which is available for + Couchbase Server Enterprise Edition only.

    @@ -204,7 +207,7 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS -h, --help - Show this help message and exit + Show this help message and exit. -B BUCKET_DESTINATION @@ -213,28 +216,28 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS -b BUCKET_SOURCE, --bucket-source=BUCKET_SOURCE - Specify the bucket to restore. Defaults to all buckets + Specify the bucket to restore. Defaults to all buckets. -u USERNAME, --username=USERNAME - REST username for source cluster or server node. Default is Administrator + REST username for source cluster or server node. Default is Administrator. -p PASSWORD, --password=PASSWORD - REST password for source cluster or server node + REST password for source cluster or server node. -s, --ssl - Transfer data with SSL enabled + Transfer data with SSL enabled. -v, --verbose - Enable verbose messaging + Enable verbose messaging. --path=PATH Specify the path to cbrestore. Defaults to current - directory + directory. --port=PORT @@ -284,11 +287,11 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS backoff_cap=10 - Max backoff time during rebalance period + Max backoff time during rebalance period. batch_max_bytes=400000 - Transfer this # of bytes per batch + Transfer this # of bytes per batch. batch_max_size=1000 @@ -350,7 +353,10 @@ cbrestorewrapper BACKUPDIR CLUSTER OPTIONS uncompress=0 - For value 1, restore data in uncompressed mode. + For value 1, restore data in uncompressed mode. +

    This option is unsupported. To restore from compressed backups, you should use + , which is available for + Couchbase Server Enterprise Edition only.

         diff --git a/content/cli/cbcli/couchbase-cli-admin-role-manage.dita b/content/cli/cbcli/couchbase-cli-admin-role-manage.dita deleted file mode 100644 index 25423030db..0000000000 --- a/content/cli/cbcli/couchbase-cli-admin-role-manage.dita +++ /dev/null @@ -1,202 +0,0 @@ - - - admin-role-manage - - Manage user roles
    SYNTAX - - - couchbase-cli admin-role-manage [--cluster <url>] [--username <user>] - [--password <password>] [--my-roles] [--get-roles] - [--set-users <user_list>] [--set-names <name_list>] [--roles] - [--delete-users <user_list>]
    DESCRIPTION - - -

    DEPRECATED: This command was in 5.0.0 deprecated and will be removed in future -releases. Please use the -couchbase-cli-user-manage1 - subcommand which -provides similar functionality to this command.

    -

    This command is used to manage LDAP user roles.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---my-roles -
    -
    -

    - Show the current users roles. -

    -
    -     - -
    ---get-roles -
    -
    -

    - Show all valid users and roles. -

    -
    -     - -
    ---set-users <user_list> -
    -
    -

    - A comma-delimited list of user ids to set acess-control roles for. -

    -
    -     - -
    ---set-names <name_list> -
    -
    -

    - A optional quoted, comma-delimited list names, one for each specified user - id. -

    -
    -     - -
    ---roles -
    -
    -

    - A comma-delimited list of roles to set for users. -

    -
    -     - -
    ---delete-users <user_list> -
    -
    -

    - A comma-delimited list of users to remove from access control. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To show the current users roles run the following command.

    - 
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
- -p password --my-roles
    -

    To see a list of all available roles in Couchbase Server run the following -command.

    - 
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
- -p password --get-roles
    -

    To set roles for a new user "Alice Meyers" with username "alice" and the -cluster administrator and replication administrator roles run the following -command.

    - 
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
- -p password --set-users alice --set-names "Alice Meyers" \
- --roles cluster_admin,replication_admin
    -

    You can also specify multiple users at the same time when using the set users -flag. For example if you want to add Alice Meyers and Barry Dillon to the same -set of roles you could run the following command.

    - 
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
- -p password --set-users alice,barry \
- --set-names "Alice Meyers,Barry Dillon" \
- --roles cluster_admin,replication_admin
    -

    To delete the user alice you can run the following command.

    - 
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
- -p password --delete-users alice
    -

    You also delete multiple users at the same time. For example if you want to -delete both alic and barry then you can do so by running the following command.

    - 
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
- -p password --delete-users alice,barry
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-admin-role-manage.html b/content/cli/cbcli/couchbase-cli-admin-role-manage.html new file mode 100644 index 0000000000..df672ae730 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-admin-role-manage.html @@ -0,0 +1,977 @@ + + + + + +couchbase-cli-admin-role-manage(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli admin-role-manage [--cluster <url>] [--username <user>]
+          [--password <password>] [--my-roles] [--get-roles]
+          [--set-users <user_list>] [--set-names <name_list>] [--roles]
+          [--delete-users <user_list>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    DEPRECATED: This command was in 5.0.0 deprecated and will be removed in future +releases. Please use the couchbase-cli-user-manage(1) subcommand which +provides similar functionality to this command.

    +

    This command is used to manage LDAP user roles.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--my-roles +
    +
    +

    + Show the current users roles. +

    +
    +
    +--get-roles +
    +
    +

    + Show all valid users and roles. +

    +
    +
    +--set-users <user_list> +
    +
    +

    + A comma-delimited list of user ids to set access-control roles for. +

    +
    +
    +--set-names <name_list> +
    +
    +

    + A optional quoted, comma-delimited list names, one for each specified user + id. +

    +
    +
    +--roles +
    +
    +

    + A comma-delimited list of roles to set for users. +

    +
    +
    +--delete-users <user_list> +
    +
    +

    + A comma-delimited list of users to remove from access control. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To show the current users roles run the following command.

    +
    +
    +
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
+ -p password --my-roles
    +
    +

    To see a list of all available roles in Couchbase Server run the following +command.

    +
    +
    +
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
+ -p password --get-roles
    +
    +

    To set roles for a new user "Alice Meyers" with username "alice" and the +cluster administrator and replication administrator roles run the following +command.

    +
    +
    +
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
+ -p password --set-users alice --set-names "Alice Meyers" \
+ --roles cluster_admin,replication_admin
    +
    +

    You can also specify multiple users at the same time when using the set users +flag. For example if you want to add Alice Meyers and Barry Dillon to the same +set of roles you could run the following command.

    +
    +
    +
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
+ -p password --set-users alice,barry \
+ --set-names "Alice Meyers,Barry Dillon" \
+ --roles cluster_admin,replication_admin
    +
    +

    To delete the user alice you can run the following command.

    +
    +
    +
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
+ -p password --delete-users alice
    +
    +

    You also delete multiple users at the same time. For example if you want to +delete both alic and barry then you can do so by running the following command.

    +
    +
    +
    $ couchbase-cli admin-role-manage -c 192.168.1.5 -u Administrator \
+ -p password --delete-users alice,barry
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-ldap(1) +couchbase-cli-user-manage(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-bucket-compact.dita b/content/cli/cbcli/couchbase-cli-bucket-compact.dita deleted file mode 100644 index f82a4dd644..0000000000 --- a/content/cli/cbcli/couchbase-cli-bucket-compact.dita +++ /dev/null @@ -1,156 +0,0 @@ - - - bucket-compact - - Compact a bucket
    SYNTAX - - - couchbase-cli bucket-compact [--cluster <url>] [--username <user>] - [--password <password>] [--bucket <name>]
    DESCRIPTION - - -

    Starts compaction of view index and data files for a specified bucket. To -compact view index files only, specify the --view-only option. To compact data -files only, specify the --data-only option. Only Couchbase buckets can be -compacted: Ephemeral and Memcached buckets cannot be compacted.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---bucket -
    -
    -

    - The name of the bucket on which to perform compaction. -

    -
    -     - -
    ---data-only -
    -
    -

    - Indicates that only data files should be compacted, and that compaction - for view index files should therefore be skipped. This option cannot be - specified at the same time as the --view-only option. -

    -
    -     - -
    ---view-only -
    -
    -

    - Indicates that only view index files should be compacted, and that - compaction for data files should therefore be skipped. This option cannot be - specified at the same time as the --data-only option. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To start compaction of both data and view index files for a bucket named -"travel-data", run the following command.

    - 
    $ couchbase-cli bucket-compact -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data
    -

    If view index files only should be compacted for a specific bucket, and -data files thereby skipped, run the following command.

    - 
    $ couchbase-cli bucket-compact -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data --view-only
    -

    If data files only should be compacted for a specific bucket, and view -index files thereby skipped, run the following command.

    - 
    $ couchbase-cli bucket-compact -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data --data-only
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-bucket-compact.html b/content/cli/cbcli/couchbase-cli-bucket-compact.html new file mode 100644 index 0000000000..355fb92a1a --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-bucket-compact.html @@ -0,0 +1,933 @@ + + + + + +couchbase-cli-bucket-compact(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli bucket-compact [--cluster <url>] [--username <user>]
+          [--password <password>] [--bucket <name>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Starts compaction of view index and data files for a specified bucket. To +compact view index files only, specify the --view-only option. To compact data +files only, specify the --data-only option. Only Couchbase buckets can be +compacted: Ephemeral and Memcached buckets cannot be compacted.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--bucket +
    +
    +

    + The name of the bucket on which to perform compaction. +

    +
    +
    +--data-only +
    +
    +

    + Indicates that only data files should be compacted, and that compaction + for view index files should therefore be skipped. This option cannot be + specified at the same time as the --view-only option. +

    +
    +
    +--view-only +
    +
    +

    + Indicates that only view index files should be compacted, and that + compaction for data files should therefore be skipped. This option cannot be + specified at the same time as the --data-only option. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To start compaction of both data and view index files for a bucket named +"travel-data", run the following command.

    +
    +
    +
    $ couchbase-cli bucket-compact -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data
    +
    +

    If view index files only should be compacted for a specific bucket, and +data files thereby skipped, run the following command.

    +
    +
    +
    $ couchbase-cli bucket-compact -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data --view-only
    +
    +

    If data files only should be compacted for a specific bucket, and view +index files thereby skipped, run the following command.

    +
    +
    +
    $ couchbase-cli bucket-compact -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data --data-only
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-bucket-compact(1). +couchbase-cli-bucket-create(1). +couchbase-cli-bucket-delete(1). +couchbase-cli-bucket-edit(1). +couchbase-cli-bucket-flush(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-bucket-create.dita b/content/cli/cbcli/couchbase-cli-bucket-create.dita deleted file mode 100644 index 368be95880..0000000000 --- a/content/cli/cbcli/couchbase-cli-bucket-create.dita +++ /dev/null @@ -1,300 +0,0 @@ - - - bucket-create - - Create a new bucket
    SYNTAX - - - couchbase-cli bucket-create [--cluster <url>] [--username <user>] - [--password <password>] [--bucket <name>] [--bucket-type <type>] - [--bucket-ramsize <size>] [--bucket-replica <num>] [--bucket-port <port>] - [--bucket-priority <priority>] [--bucket-eviction-policy <policy>] - [--enable-flush <num>] [--enable-index-replica <num>] - [--conflict-resolution <type>] [--wait]
    DESCRIPTION - - -

    Creates a new bucket. Allows creation of Couchbase, Ephemeral, and Memcached -buckets, and supports the various configuration parameters for these -buckets.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---bucket <name> -
    -
    -

    - The name of the bucket to create. The only characters that can be used - for the bucket-name are those in the ranges of A-Z, a-z, and 0-9; plus the - underscore, period, dash, and percent characters. The name can be a - maximum of 100 characters in length. -

    -
    -     - -
    ---bucket-type <type> -
    -
    -

    - The type of bucket to create. Accepted bucket types are "couchbase", - "ephemeral", and "memcached". The Couchbase bucket is the standard bucket - type. It supports data persistence, replication, caching, indexing, views, - and N1QL queries. The Ephemeral bucket is an in-memory bucket similar to the - Couchbase bucket; but it does not support data persistence or views. The - Memcached bucket is a cache-only bucket that does not support persistence, - replication, indexing, views, or N1QL querying: this bucket type provides - the same behavior as Memcached Server. -

    -
    -     - -
    ---bucket-ramsize <size> -
    -
    -

    - The amount of memory to allocate to the cache for this bucket, in Megabytes. - The memory quota of this bucket must fit into the overall cluster memory - quota. The minimum cache size is 100MB. -

    -
    -     - -
    ---bucket-replica <num> -
    -
    -

    - The number of servers to which data is replicated. Replicas provide - protection against data loss by keeping copies of this bucket’s data on - multiple servers. By default, the number of replicas is one, even if there - is only a single server in the cluster. The minimum number of replicas is - zero, and the maximum three. This option is valid for Couchbase and - Ephemeral buckets only. -

    -
    -     - -
    ---bucket-port <num> -
    -
    -

    - Sets the port on which the bucket listens. This parameter is deprecated, and - therefore not recommended for use. -

    -
    -     - -
    ---bucket-priority <priority> -
    -
    -

    - Specifies the priority of this bucket’s background tasks. This option is - valid for Couchbase and Ephemeral buckets only. For Couchbase buckets, - background task-types include disk I/O, DCP stream-processing, and - item-paging. For Ephemeral buckets, background task-types are the same as - for Couchbase buckets, with the exception of disk I/O, which does not apply - to Ephemeral buckets. The value of this option may be "high" or "low". The - default is "low". Specifying "high" may result in faster processing; but - only when more than one bucket is defined for the cluster, and when - different priority settings have been established among the buckets. When - Couchbase and Ephemeral buckets have different priority settings, this - affects the prioritization only of task-types other than disk I/O. -

    -
    -     - -
    ---bucket-eviction-policy <policy> -
    -
    -

    - The memory-cache eviction policy for this bucket. This option is valid for - Couchbase and Ephemeral buckets only. -

    -
    Couchbase buckets support either "valueOnly" or "fullEviction". Specifying
-the "valueOnly" policy means that each key stored in this bucket must be
-kept in memory. This is the default policy: using this policy improves
-performance of key-value operations, but limits the maximum size of the
-bucket. Specifying the "fullEviction" policy means that performance is
-impacted for key-value operations, but the maximum size of the bucket is
-unbounded.
    -
    Ephemeral buckets support either "noEviction" or "nruEviction". Specifying
-"noEviction" means that the bucket will not evict items from the cache if
-the cache is full: this type of eviction policy should be used for in-memory
-database use-cases. Specifying "nruEviction" means that items not recently
-used will be evicted from memory, when all memory in the bucket is used:
-this type of eviction policy should be used for caching use-cases.
    -
    -     - -
    ---enable-flush <num> -
    -
    -

    - Specifies whether or not the flush operation is allowed for this bucket. To - enable flushing, set this option to "1". To disable flushing, set this - option to "0". By default, flushing is disabled. -

    -
    -     - -
    ---enable-index-replica <num> -
    -
    -

    - Enables view index replication for the current bucket. This option is valid - for Couchbase buckets only. To enable, set the value of this option to "1". - To disable, set the value of this option to "0". By default, view index - replication is disabled. There may be at most one replica view index. -

    -
    -     - -
    ---conflict-resolution <type> -
    -
    -

    - Specifies this bucket’s conflict resolution mechanism; which is to be used - if a conflict occurs during Cross Data-Center Replication (XDCR). - Sequence-based and timestamp-based mechanisms are supported. -

    -
        Sequence-based conflict resolution selects the document that has been
-    updated the greatest number of times since the last sync: for example, if
-    one cluster has updated a document twice since the last sync, and the other
-    cluster has updated the document three times, the document updated three
-    times is selected; regardless of the specific times at which updates took
-place.
    -
    Timestamp-based conflict resolution selects the document with the most
-recent timestamp: this is only supported when all clocks on all
-cluster-nodes have been fully synchronized.
    -
    -     -
    -

    --wait - The create bucket command is asynchronous by default. Specifying this option - makes the operation synchronous: so that the command returns only after the - bucket has been fully created.

    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To create a Couchbase bucket named "travel-data" with a memory cache size -of 1GB, run the following command.

    - 
    $ couchbase-cli bucket-create -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data --bucket-type couchbase \
- --bucket-ramsize 1024
    -

    To create a Couchbase bucket named "airline-data" with a memory cache size of -1GB, two data replicas, high background-task priority, full eviction, flushing -enabled, and view index replication enabled, run the following command.

    - 
    $ couchbase-cli bucket-create -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data --bucket-type couchbase \
- --bucket-ramsize 1024 --bucket-replica 2 --bucket-priority high \
- --bucket-eviction-policy fullEviction --enable-flush 1 \
- --enable-index-replica 1
    -

    To create a Memcached bucket named "travel-data" with a memory cache size -of 1GB, run the following command.

    - 
    $ couchbase-cli bucket-create -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data --bucket-type memcached \
- --bucket-ramsize 1024
    -

    To create an Ephemeral bucket named "hotel-data" synchronously, with a memory -cache size of 256MB, flushing enabled, "nruEviction", and timestamp-based -conflict resolution, run the following command.

    - 
    $ couchbase-cli bucket-create -c 192.168.1.5:8091 --username Administrator \
---password password --bucket hotel-data --bucket-type ephemeral \
---bucket-ramsize 256 --enable-flush 1 --bucket-eviction-policy nruEviction \
---conflict-resolution timestamp --wait
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-bucket-create.html b/content/cli/cbcli/couchbase-cli-bucket-create.html new file mode 100644 index 0000000000..0b6aedc84b --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-bucket-create.html @@ -0,0 +1,1116 @@ + + + + + +couchbase-cli-bucket-create(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli bucket-create [--cluster <url>] [--username <user>]
+          [--password <password>] [--bucket <name>] [--bucket-type <type>]
+          [--bucket-ramsize <size>] [--bucket-replica <num>]
+          [--bucket-priority <priority>] [--bucket-eviction-policy <policy>]
+          [--max-ttl <seconds>] [--compression-mode <mode>] [--enable-flush <num>]
+          [--enable-index-replica <num>] [--conflict-resolution <type>] [--wait]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Creates a new bucket. Allows creation of Couchbase, Ephemeral, and Memcached +buckets, and supports the various configuration parameters for these +buckets.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--bucket <name> +
    +
    +

    + The name of the bucket to create. The only characters that can be used + for the bucket-name are those in the ranges of A-Z, a-z, and 0-9; plus the + underscore, period, dash, and percent characters. The name can be a + maximum of 100 characters in length. +

    +
    +
    +--bucket-type <type> +
    +
    +

    + The type of bucket to create. Accepted bucket types are "couchbase", + "ephemeral", and "memcached". The Couchbase bucket is the standard bucket + type. It supports data persistence, replication, caching, indexing, views, + and N1QL queries. The Ephemeral bucket is an in-memory bucket similar to the + Couchbase bucket; but it does not support data persistence or views. The + Memcached bucket is a cache-only bucket that does not support persistence, + replication, indexing, views, or N1QL querying: this bucket type provides + the same behavior as Memcached Server. +

    +
    +
    +--bucket-ramsize <size> +
    +
    +

    + The amount of memory to allocate to the cache for this bucket, in Megabytes. + The memory quota of this bucket must fit into the overall cluster memory + quota. The minimum cache size is 100MB. +

    +
    +
    +--bucket-replica <num> +
    +
    +

    + The number of servers to which data is replicated. Replicas provide + protection against data loss by keeping copies of this bucket’s data on + multiple servers. By default, the number of replicas is one, even if there + is only a single server in the cluster. The minimum number of replicas is + zero, and the maximum three. This option is valid for Couchbase and + Ephemeral buckets only. +

    +
    +
    +--bucket-priority <priority> +
    +
    +

    + Specifies the priority of this bucket’s background tasks. This option is + valid for Couchbase and Ephemeral buckets only. For Couchbase buckets, + background task-types include disk I/O, DCP stream-processing, and + item-paging. For Ephemeral buckets, background task-types are the same as + for Couchbase buckets, with the exception of disk I/O, which does not apply + to Ephemeral buckets. The value of this option may be "high" or "low". The + default is "low". Specifying "high" may result in faster processing; but + only when more than one bucket is defined for the cluster, and when + different priority settings have been established among the buckets. When + Couchbase and Ephemeral buckets have different priority settings, this + affects the prioritization only of task-types other than disk I/O. +

    +
    +
    +--bucket-eviction-policy <policy> +
    +
    +

    + The memory-cache eviction policy for this bucket. This option is valid for + Couchbase and Ephemeral buckets only. +

    +
    +
    +
    Couchbase buckets support either "valueOnly" or "fullEviction". Specifying
+the "valueOnly" policy means that each key stored in this bucket must be
+kept in memory. This is the default policy: using this policy improves
+performance of key-value operations, but limits the maximum size of the
+bucket. Specifying the "fullEviction" policy means that performance is
+impacted for key-value operations, but the maximum size of the bucket is
+unbounded.
    +
    +
    +
    +
    Ephemeral buckets support either "noEviction" or "nruEviction". Specifying
+"noEviction" means that the bucket will not evict items from the cache if
+the cache is full: this type of eviction policy should be used for in-memory
+database use-cases. Specifying "nruEviction" means that items not recently
+used will be evicted from memory, when all memory in the bucket is used:
+this type of eviction policy should be used for caching use-cases.
    +
    +
    +
    +--enable-flush <num> +
    +
    +

    + Specifies whether or not the flush operation is allowed for this bucket. To + enable flushing, set this option to "1". To disable flushing, set this + option to "0". By default, flushing is disabled. +

    +
    +
    +--enable-index-replica <num> +
    +
    +

    + Enables view index replication for the current bucket. This option is valid + for Couchbase buckets only. To enable, set the value of this option to "1". + To disable, set the value of this option to "0". By default, view index + replication is disabled. There may be at most one replica view index. +

    +
    +
    +--conflict-resolution <type> +
    +
    +

    + Specifies this bucket’s conflict resolution mechanism; which is to be used + if a conflict occurs during Cross Data-Center Replication (XDCR). + Sequence-based and timestamp-based mechanisms are supported. +

    +
    +
    +
        Sequence-based conflict resolution selects the document that has been
+    updated the greatest number of times since the last sync: for example, if
+    one cluster has updated a document twice since the last sync, and the other
+    cluster has updated the document three times, the document updated three
+    times is selected; regardless of the specific times at which updates took
+place.
    +
    +
    +
    +
    Timestamp-based conflict resolution selects the document with the most
+recent timestamp: this is only supported when all clocks on all
+cluster-nodes have been fully synchronized.
    +
    +
    +
    +--max-ttl <seconds> +
    +
    +

    + Specifies the maximum TTL (time-to-live) for all documents in + bucket in seconds. If enabled and a document is mutated with no TTL + or a TTL greater than than the maximum, its TTL will be set to the + maximum TTL. Setting this option to 0 disables the use of max-TTL + and the largest TTL that is allowed is 2147483647. This option is + only available for Couchbase and Ephemeral buckets in Couchbase + Enterprise Edition. +

    +
    +
    +--compression-mode <mode> +
    +
    +

    + Specifies the compression-mode of the bucket. There are three options; + off, passive and active. All three modes are backwards compatible with + older SDKs, where Couchbase Server will automatically uncompress documents + for clients that do not understand the compression being used. This option + is only available for Couchbase and Ephemeral buckets in Couchbase + Enterprise Edition. +

    +
    +
    +
    Off: Couchbase Server will only compress document values when persisting to
+disk. This is suitable for use cases where compression could have a
+negative impact on performance. Please note it is expected that compression
+in most use cases will improve performance.
    +
    +
    +
    +
    Passive: Documents which were compressed by a client, or read compressed
+from disk will be stored compress in-memory. Couchbase Server will make
+no additional attempt to compress documents that are not already compressed.
    +
    +
    +
    +
    Active: Couchbase Server will actively and aggressively attempt to compress
+documents, even if they have not been received in a compressed format. This
+provides the benefits of compression even when the SDK clients are not
+complicit.
    +
    +
    +
    +

    --wait + The create bucket command is asynchronous by default. Specifying this option + makes the operation synchronous: so that the command returns only after the + bucket has been fully created.

    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To create a Couchbase bucket named "travel-data" with a memory cache size +of 1GB, run the following command.

    +
    +
    +
    $ couchbase-cli bucket-create -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data --bucket-type couchbase \
+ --bucket-ramsize 1024
    +
    +

    To create a Couchbase bucket named "airline-data" with a memory cache size of +1GB, two data replicas, high background-task priority, full eviction, flushing +enabled, and view index replication enabled, run the following command.

    +
    +
    +
    $ couchbase-cli bucket-create -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data --bucket-type couchbase \
+ --bucket-ramsize 1024 --bucket-replica 2 --bucket-priority high \
+ --bucket-eviction-policy fullEviction --enable-flush 1 \
+ --enable-index-replica 1
    +
    +

    To create a Memcached bucket named "travel-data" with a memory cache size +of 1GB, run the following command.

    +
    +
    +
    $ couchbase-cli bucket-create -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data --bucket-type memcached \
+ --bucket-ramsize 1024
    +
    +

    To create an Ephemeral bucket named "hotel-data" synchronously, with a memory +cache size of 256MB, flushing enabled, "nruEviction", and timestamp-based +conflict resolution, run the following command.

    +
    +
    +
    $ couchbase-cli bucket-create -c 192.168.1.5:8091 --username Administrator \
+--password password --bucket hotel-data --bucket-type ephemeral \
+--bucket-ramsize 256 --enable-flush 1 --bucket-eviction-policy nruEviction \
+--conflict-resolution timestamp --wait
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-bucket-compact(1). +couchbase-cli-bucket-delete(1). +couchbase-cli-bucket-edit(1). +couchbase-cli-bucket-flush(1). +couchbase-cli-bucket-list(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-bucket-delete.dita b/content/cli/cbcli/couchbase-cli-bucket-delete.dita deleted file mode 100644 index bb7c1452c0..0000000000 --- a/content/cli/cbcli/couchbase-cli-bucket-delete.dita +++ /dev/null @@ -1,121 +0,0 @@ - - - bucket-delete - - Delete a bucket
    SYNTAX - - - couchbase-cli bucket-delete [--cluster <url>] [--username <user>] - [--password <password>] [--bucket <name>]
    DESCRIPTION - - -

    Deletes a bucket in the cluster. When the bucket is deleted, all indexes on the -bucket are also deleted.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---bucket <name> -
    -
    -

    - The name of the bucket to delete. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To delete a bucket named "travel-data", run the command below.

    - 
    $ couchbase-cli bucket-delete -c 192.168.1.5 --username Administrator \
- --password password --bucket travel-data
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-bucket-delete.html b/content/cli/cbcli/couchbase-cli-bucket-delete.html new file mode 100644 index 0000000000..e1a2f64d99 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-bucket-delete.html @@ -0,0 +1,896 @@ + + + + + +couchbase-cli-bucket-delete(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli bucket-delete [--cluster <url>] [--username <user>]
+          [--password <password>] [--bucket <name>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Deletes a bucket in the cluster. When the bucket is deleted, all indexes on the +bucket are also deleted.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--bucket <name> +
    +
    +

    + The name of the bucket to delete. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To delete a bucket named "travel-data", run the command below.

    +
    +
    +
    $ couchbase-cli bucket-delete -c 192.168.1.5 --username Administrator \
+ --password password --bucket travel-data
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-bucket-compact(1). +couchbase-cli-bucket-create(1). +couchbase-cli-bucket-edit(1). +couchbase-cli-bucket-flush(1). +couchbase-cli-bucket-list(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-bucket-edit.dita b/content/cli/cbcli/couchbase-cli-bucket-edit.dita deleted file mode 100644 index b5bb4bb260..0000000000 --- a/content/cli/cbcli/couchbase-cli-bucket-edit.dita +++ /dev/null @@ -1,240 +0,0 @@ - - - bucket-edit - - Edit a bucket
    SYNTAX - - - couchbase-cli bucket-edit [--cluster <url>] [--username <user>] - [--password <password>] [--bucket <name>] [--bucket-ramsize <size>] - [--bucket-replica <num>] [--bucket-priority <priority>] - [--bucket-eviction-policy <policy>] [--enable-flush <num>]
    DESCRIPTION - - -

    Modifies the settings of the bucket specified. Note that some settings can be -applied immediately, while other settings either require a rebalance, or require -the bucket to be restarted; resulting in potential application downtime.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---bucket <name> -
    -
    -

    - The name of the bucket to edit. -

    -
    -     - -
    ---bucket-ramsize <size> -
    -
    -

    - The amount of memory to allocate to the cache for this bucket, in Megabytes. - The memory quota of this bucket must fit into the overall cluster memory - quota. The minimum cache size is 100MB. -

    -
    -     - -
    ---bucket-replica <num> -
    -
    -

    - The number of servers to which data is replicated. Replicas provide - protection against data loss by keeping copies of a bucket’s data on - multipleservers. By default, the number of replicas is one, even if there is - only a single server in the cluster. The minimum number of replicas is zero, - and the maximum three. This option is valid for Couchbase and Ephemeral - buckets only. -

    -
    -     - -
    ---bucket-port <num> -
    -
    -

    - Sets the port on which the bucket listens. This parameter is deprecated, and - therefore not recommended for use. -

    -
    -     - -
    ---bucket-priority <priority> -
    -
    -

    - Specifies the priority of this bucket’s background tasks. This option is - valid for Couchbase and Ephemeral buckets only. For Couchbase buckets, - background task-types include disk I/O, DCP stream-processing, and - item-paging. For Ephemeral buckets, background task-types are the same as - for Couchbase buckets, with the exception of disk I/O, which does not apply - to Ephemeral buckets. The value of this option may be "high" or "low". The - default is "low". Specifying "high" may result in faster processing; but - only when more than one bucket is defined for the cluster, and when - different priority settings have been established among the buckets. When - Couchbase and Ephemeral buckets have different priority settings, this - affects the prioritization only of task-types other than disk I/O. -

    -
    -     - -
    ---bucket-eviction-policy <policy> -
    -
    -

    - The memory-cache eviction policy for this bucket. This option is valid for - Couchbase and Ephemeral buckets only. -

    -
    Couchbase buckets support either "valueOnly" or "fullEviction". Specifying
-the "valueOnly" policy means that each key stored in this bucket must be
-kept in memory. This is the default policy: using this policy improves
-performance of key-value operations, but limits the maximum size of the
-bucket. Specifying the "fullEviction" policy means that performance is
-impacted for key-value operations, but the maximum size of the bucket is
-unbounded.
    -
    Ephemeral buckets support either "noEviction" or "nruEviction". Specifying
-"noEviction" means that the bucket will not evict items from the cache if
-the cache is full: this type of eviction policy should be used for in-memory
-database use-cases. Specifying "nruEviction" means that items not recently
-used will be evicted from memory, when all memory in the bucket is used:
-this type of eviction policy should be used for caching use-cases.
    -
    -     - -
    ---enable-flush <num> -
    -
    -

    - Specifies whether or not the flush operation is allowed for this bucket. To - enable flushing, set this option to "1". To disable flushing, set this - option to "0". By default, flushing is disabled. -

    -
    -     - -
    ---enable-index-replica <num> -
    -
    -

    - Enables view index replication for the current bucket. This option is valid - for Couchbase buckets only. To enable, set the value of this option to "1". - To disable, set the value of this option to "0". By default, view index - replication is disabled. There may be at most one replica view index. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To change the memory quota of the travel-data bucket, run the following -command:

    - 
    $ couchbase-cli bucket-edit -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data --bucket-ramsize 1024
    -

    To change the number of replicas for the travel-data bucket to "2", run the -following command. (Note that this requires a subsequent rebalance, by means of - -couchbase-cli-rebalance1 -, to ensure that the replicas are created.)

    - 
    $ couchbase-cli bucket-edit -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data --bucket-ramsize 1024 \
- --bucket-replicas 2
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-bucket-edit.html b/content/cli/cbcli/couchbase-cli-bucket-edit.html new file mode 100644 index 0000000000..7e3c00e29f --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-bucket-edit.html @@ -0,0 +1,1076 @@ + + + + + +couchbase-cli-bucket-edit(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli bucket-edit [--cluster <url>] [--username <user>]
+          [--password <password>] [--bucket <name>] [--bucket-ramsize <size>]
+          [--bucket-replica <num>] [--bucket-priority <priority>]
+          [--bucket-eviction-policy <policy>] [--enable-flush <num>]
+          [--max-ttl <seconds>] [--compression-mode <mode>]
+          [--remove-bucket-port <num>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Modifies the settings of the bucket specified. Note that some settings can be +applied immediately, while other settings either require a rebalance, or require +the bucket to be restarted; resulting in potential application downtime.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--bucket <name> +
    +
    +

    + The name of the bucket to edit. +

    +
    +
    +--bucket-ramsize <size> +
    +
    +

    + The amount of memory to allocate to the cache for this bucket, in Megabytes. + The memory quota of this bucket must fit into the overall cluster memory + quota. The minimum cache size is 100MB. +

    +
    +
    +--bucket-replica <num> +
    +
    +

    + The number of servers to which data is replicated. Replicas provide + protection against data loss by keeping copies of a bucket’s data on + multiple servers. By default, the number of replicas is one, even if there is + only a single server in the cluster. The minimum number of replicas is zero, + and the maximum three. This option is valid for Couchbase and Ephemeral + buckets only. +

    +
    +
    +--bucket-port <num> +
    +
    +

    + Sets the port on which the bucket listens. This parameter is deprecated, and + therefore not recommended for use. +

    +
    +
    +--bucket-priority <priority> +
    +
    +

    + Specifies the priority of this bucket’s background tasks. This option is + valid for Couchbase and Ephemeral buckets only. For Couchbase buckets, + background task-types include disk I/O, DCP stream-processing, and + item-paging. For Ephemeral buckets, background task-types are the same as + for Couchbase buckets, with the exception of disk I/O, which does not apply + to Ephemeral buckets. The value of this option may be "high" or "low". The + default is "low". Specifying "high" may result in faster processing; but + only when more than one bucket is defined for the cluster, and when + different priority settings have been established among the buckets. When + Couchbase and Ephemeral buckets have different priority settings, this + affects the prioritization only of task-types other than disk I/O. +

    +
    +
    +--bucket-eviction-policy <policy> +
    +
    +

    + The memory-cache eviction policy for this bucket. This option is valid for + Couchbase and Ephemeral buckets only. +

    +
    +
    +
    Couchbase buckets support either "valueOnly" or "fullEviction". Specifying
+the "valueOnly" policy means that each key stored in this bucket must be
+kept in memory. This is the default policy: using this policy improves
+performance of key-value operations, but limits the maximum size of the
+bucket. Specifying the "fullEviction" policy means that performance is
+impacted for key-value operations, but the maximum size of the bucket is
+unbounded.
    +
    +
    +
    +
    Ephemeral buckets support either "noEviction" or "nruEviction". Specifying
+"noEviction" means that the bucket will not evict items from the cache if
+the cache is full: this type of eviction policy should be used for in-memory
+database use-cases. Specifying "nruEviction" means that items not recently
+used will be evicted from memory, when all memory in the bucket is used:
+this type of eviction policy should be used for caching use-cases.
    +
    +
    +
    +--enable-flush <num> +
    +
    +

    + Specifies whether or not the flush operation is allowed for this bucket. To + enable flushing, set this option to "1". To disable flushing, set this + option to "0". By default, flushing is disabled. +

    +
    +
    +--enable-index-replica <num> +
    +
    +

    + Enables view index replication for the current bucket. This option is valid + for Couchbase buckets only. To enable, set the value of this option to "1". + To disable, set the value of this option to "0". By default, view index + replication is disabled. There may be at most one replica view index. +

    +
    +
    +--max-ttl <seconds> +
    +
    +

    + Specifies the maximum TTL (time-to-live) for all documents in + bucket in seconds. If enabled and a document is mutated with no TTL + or a TTL greater than than the maximum, its TTL will be set to the + maximum TTL. Setting this option to 0 disables the use of max-TTL + and the largest TTL that is allowed is 2147483647. This option is + only available for Couchbase and Ephemeral buckets in Couchbase + Enterprise Edition. +

    +
    +
    +--compression-mode <mode> +
    +
    +

    + Specifies the compression-mode of the bucket. There are three options; + off, passive and active. All three modes are backwards compatible with + older SDKs, where Couchbase Server will automatically uncompress documents + for clients that do not understand the compression being used. This option + is only available for Couchbase and Ephemeral buckets in Couchbase + Enterprise Edition. +

    +
    +
    +
    Off: Couchbase Server will only compress document values when persisting to
+disk. This is suitable for use cases where compression could have a
+negative impact on performance. Please note it is expected that compression
+in most use cases will improve performance.
    +
    +
    +
    +
    Passive: Documents which were compressed by a client, or read compressed
+from disk will be stored compress in-memory. Couchbase Server will make
+no additional attempt to compress documents that are not already compressed.
    +
    +
    +
    +
    Active: Couchbase Server will actively and aggressively attempt to compress
+documents, even if they have not been received in a compressed format. This
+provides the benefits of compression even when the SDK clients are not
+complicit.
    +
    +
    +
    +--remove-bucket-port <num> +
    +
    +

    + Removes the bucket-port setting that might have been set in previous + versions of Couchbase Server. The bucket-port also knows as the dedicated + port and server side moxi has been deprecated. In later versions of + Couchbase Server a cluster will not be able to upgrade if a bucket has the + dedicated port set. This option allows removal of that setting by setting + it to 1. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To change the memory quota of the travel-data bucket, run the following +command:

    +
    +
    +
    $ couchbase-cli bucket-edit -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data --bucket-ramsize 1024
    +
    +

    To change the number of replicas for the travel-data bucket to "2", run the +following command. (Note that this requires a subsequent rebalance, by means of +couchbase-cli-rebalance(1), to ensure that the replicas are created.)

    +
    +
    +
    $ couchbase-cli bucket-edit -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data --bucket-ramsize 1024 \
+ --bucket-replicas 2
    +
    +

    To remove the bucket-port from the test bucket

    +
    +
    +
    $ couchbase-cli bucket-edit -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket test --remove-bucket-port 1
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-bucket-compact(1). +couchbase-cli-bucket-create(1). +couchbase-cli-bucket-delete(1). +couchbase-cli-bucket-flush(1). +couchbase-cli-bucket-list(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-bucket-flush.dita b/content/cli/cbcli/couchbase-cli-bucket-flush.dita deleted file mode 100644 index 1719ef369f..0000000000 --- a/content/cli/cbcli/couchbase-cli-bucket-flush.dita +++ /dev/null @@ -1,122 +0,0 @@ - - - bucket-flush - - Flush a bucket
    SYNTAX - - - couchbase-cli bucket-flush [--cluster <url>] [--username <user>] - [--password <password>] [--bucket <name>]
    DESCRIPTION - - -

    Deletes all data from a bucket. The bucket itself is not deleted; and all -its indexes and configuration-details remain unchanged. Applies to -Couchbase, Ephemeral, and Memcached buckets.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---bucket <name> -
    -
    -

    - The name of the bucket to flush. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To flush a bucket named "travel-data", run the following command.

    - 
    $ couchbase-cli bucket-flush -c 192.168.1.5:8091 --username Administrator \
- --password password --bucket travel-data
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-bucket-flush.html b/content/cli/cbcli/couchbase-cli-bucket-flush.html new file mode 100644 index 0000000000..4778246d57 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-bucket-flush.html @@ -0,0 +1,897 @@ + + + + + +couchbase-cli-bucket-flush(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli bucket-flush [--cluster <url>] [--username <user>]
+          [--password <password>] [--bucket <name>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Deletes all data from a bucket. The bucket itself is not deleted; and all +its indexes and configuration-details remain unchanged. Applies to +Couchbase, Ephemeral, and Memcached buckets.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--bucket <name> +
    +
    +

    + The name of the bucket to flush. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To flush a bucket named "travel-data", run the following command.

    +
    +
    +
    $ couchbase-cli bucket-flush -c 192.168.1.5:8091 --username Administrator \
+ --password password --bucket travel-data
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-bucket-compact(1). +couchbase-cli-bucket-create(1). +couchbase-cli-bucket-delete(1). +couchbase-cli-bucket-edit(1). +couchbase-cli-bucket-list(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-bucket-list.dita b/content/cli/cbcli/couchbase-cli-bucket-list.dita deleted file mode 100644 index af8e77021a..0000000000 --- a/content/cli/cbcli/couchbase-cli-bucket-list.dita +++ /dev/null @@ -1,134 +0,0 @@ - - - bucket-list - - List all buckets
    SYNTAX - - - couchbase-cli bucket-list [--cluster <url>] [--username <user>] - [--password <password>]
    DESCRIPTION - - -

    Lists all buckets in the cluster. For each bucket, the following are printed -to stdout: bucketType, authType, numReplicas, ramQuota, ramUsed.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To list all buckets in the cluster, run the following command.

    - 
    $ couchbase-cli bucket-list -c 192.168.1.5:8091 --username Administrator \
- --password password
    - 
    travel-data
- bucketType: membase
- authType: sasl
- numReplicas: 1
- ramQuota: 104857600
- ramUsed: 45302840
-airline-data
- bucketType: ephemeral
- authType: sasl
- numReplicas: 1
- ramQuota: 268435456
- ramUsed: 1134432
-flight-data
- bucketType: memcached
- authType: sasl
- numReplicas: 0
- ramQuota: 104857600
- ramUsed: 0
    -
    DISCUSSION - - -

    In the output, RAM figures are in bytes. The "membase" bucketType signifies a -Couchbase bucket. The authType can be either "sasl" or "none".

    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-bucket-list.html b/content/cli/cbcli/couchbase-cli-bucket-list.html new file mode 100644 index 0000000000..346e9e0aaa --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-bucket-list.html @@ -0,0 +1,916 @@ + + + + + +couchbase-cli-bucket-list(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli bucket-list [--cluster <url>] [--username <user>]
+          [--password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Lists all buckets in the cluster. For each bucket, the following are printed +to stdout: bucketType, authType, numReplicas, ramQuota, ramUsed.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To list all buckets in the cluster, run the following command.

    +
    +
    +
    $ couchbase-cli bucket-list -c 192.168.1.5:8091 --username Administrator \
+ --password password
    +
    +
    +
    +
    travel-data
+ bucketType: membase
+ authType: sasl
+ numReplicas: 1
+ ramQuota: 104857600
+ ramUsed: 45302840
+airline-data
+ bucketType: ephemeral
+ authType: sasl
+ numReplicas: 1
+ ramQuota: 268435456
+ ramUsed: 1134432
+flight-data
+ bucketType: memcached
+ authType: sasl
+ numReplicas: 0
+ ramQuota: 104857600
+ ramUsed: 0
    +
    +
    +
    +
    +

    DISCUSSION

    +
    +

    In the output, RAM figures are in bytes. The "membase" bucketType signifies a +Couchbase bucket. The authType can be either "sasl" or "none".

    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-bucket-compact(1). +couchbase-cli-bucket-create(1). +couchbase-cli-bucket-delete(1). +couchbase-cli-bucket-edit(1). +couchbase-cli-bucket-flush(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-cluster-edit.dita b/content/cli/cbcli/couchbase-cli-cluster-edit.dita deleted file mode 100644 index 7d7b418f9c..0000000000 --- a/content/cli/cbcli/couchbase-cli-cluster-edit.dita +++ /dev/null @@ -1,216 +0,0 @@ - - - cluster-edit - - Edits cluster settings
    SYNTAX - - - couchbase-cli cluster-edit [--cluster <url>] [--cluster-username <username>] - [--cluster-password <password>] [--cluster-port <port>] - [--cluster-ramsize <megabytes>] [--cluster-name <name>] - [--cluster-index-ramsize <megabytes>] [--cluster-fts-ramsize <megabytes>]
    DESCRIPTION - - -

    DEPRECATED: Please use the -couchbase-cli-setting-cluster1 - command -which provides the same functionality as this command.

    -

    This command is used to modify cluster level settings. It allows users to change -the Couchbase Server built-in administrator username and password, change the port -that the cluster manager listens on, and modify the data, index, and full-text -service memory quotas.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---cluster-username -
    -
    -

    - Specifies the new username for the Couchbase Server administrator user. -

    -
    -     - -
    ---cluster-password -
    -
    -

    - Specifies the new password for the Couchbase Server administrator user. -

    -
    -     - -
    ---cluster-ramsize -
    -
    -

    - Sets the data service memory quota (in MB). This quota will be assigned - to all future nodes added to the cluster with the data service. -

    -
    -     - -
    ---cluster-fts-ramsize -
    -
    -

    - Sets the full-text service memory quota (in MB). This parameter is required - if the full-text service is specified in the --services option. This quota - will be assigned to all future nodes added to the cluster with the full-text - service. -

    -
    -     - -
    ---cluster-index-ramsize -
    -
    -

    - Sets the index service memory quota (in MB). This parameter is required if - the index service is specified in the --services option. This quota will be - assigned to all future nodes added to the cluster with the index service. -

    -
    -     - -
    ---cluster-name -
    -
    -

    - Sets the name for this cluster. Naming clusters is useful when you have - multiple Couchbase Server clusters in your deployment. -

    -
    -     - -
    ---cluster-port -
    -
    -

    - Specifies the port for the Couchbase Server Administration Console. Defaults - to port 8091. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To change the username and password of the Couchbase Server administrator user run -the following command.

    - 
    $ couchbase-cli cluster-edit -c 192.168.1.5:8091 --username current_user \
- --password current_password --cluster-username new_username \
- --cluster-password new_password
    -

    To change the port number that the cluster manager listens on and the cluster -name to "new_name" run the following command.

    - 
    $ couchbase-cli cluster-edit -c 192.168.1.5:8091 --username Administrator \
- --password password --cluster-port 5000 --cluster-name new_name
    -

    To change the memory quota of the data service to 2048MB and the memory quota of -the index service to 4096MB run the following command.

    - 
    $ couchbase-cli cluster-edit -c 192.168.1.5:8091 --username Administrator \
- --password password --cluster-ramsize 2048 --cluster-index-ramsize 4096
    -

    All of the parameters in this command can be specified at the same time. To -change the username and password of the Couchbase Server administrator user, change -the port number to 5000, change the cluster name to "new_name", change the memory -quota of the data service to 2048MB and change the memory quota of the index -service to 4096MB run the following command.

    - 
    $ couchbase-cli cluster-edit -c 192.168.1.5:8091 --username current_user \
- --password current_password --cluster-username new_username \
- --cluster-password new_password --cluster-port 5000 \
- --cluster-name new_name --cluster-ramsize 2048 --cluster-index-ramsize 4096
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-cluster-edit.html b/content/cli/cbcli/couchbase-cli-cluster-edit.html new file mode 100644 index 0000000000..ca6f646b76 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-cluster-edit.html @@ -0,0 +1,983 @@ + + + + + +couchbase-cli-cluster-edit(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli cluster-edit [--cluster <url>] [--cluster-username <username>]
+          [--cluster-password <password>] [--cluster-port <port>]
+          [--cluster-ramsize <megabytes>] [--cluster-name <name>]
+          [--cluster-index-ramsize <megabytes>] [--cluster-fts-ramsize <megabytes>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    DEPRECATED: Please use the couchbase-cli-setting-cluster(1) command +which provides the same functionality as this command.

    +

    This command is used to modify cluster level settings. It allows users to change +the Couchbase Server built-in administrator username and password, change the port +that the cluster manager listens on, and modify the data, index, and full-text +service memory quotas.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--cluster-username +
    +
    +

    + Specifies the new username for the Couchbase Server administrator user. +

    +
    +
    +--cluster-password +
    +
    +

    + Specifies the new password for the Couchbase Server administrator user. +

    +
    +
    +--cluster-ramsize +
    +
    +

    + Sets the data service memory quota (in MB). This quota will be assigned + to all future nodes added to the cluster with the data service. +

    +
    +
    +--cluster-fts-ramsize +
    +
    +

    + Sets the full-text service memory quota (in MB). This parameter is required + if the full-text service is specified in the --services option. This quota + will be assigned to all future nodes added to the cluster with the full-text + service. +

    +
    +
    +--cluster-index-ramsize +
    +
    +

    + Sets the index service memory quota (in MB). This parameter is required if + the index service is specified in the --services option. This quota will be + assigned to all future nodes added to the cluster with the index service. +

    +
    +
    +--cluster-name +
    +
    +

    + Sets the name for this cluster. Naming clusters is useful when you have + multiple Couchbase Server clusters in your deployment. +

    +
    +
    +--cluster-port +
    +
    +

    + Specifies the port for the Couchbase Server Administration Console. Defaults + to port 8091. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To change the username and password of the Couchbase Server administrator user run +the following command.

    +
    +
    +
    $ couchbase-cli cluster-edit -c 192.168.1.5:8091 --username current_user \
+ --password current_password --cluster-username new_username \
+ --cluster-password new_password
    +
    +

    To change the port number that the cluster manager listens on and the cluster +name to "new_name" run the following command.

    +
    +
    +
    $ couchbase-cli cluster-edit -c 192.168.1.5:8091 --username Administrator \
+ --password password --cluster-port 5000 --cluster-name new_name
    +
    +

    To change the memory quota of the data service to 2048MB and the memory quota of +the index service to 4096MB run the following command.

    +
    +
    +
    $ couchbase-cli cluster-edit -c 192.168.1.5:8091 --username Administrator \
+ --password password --cluster-ramsize 2048 --cluster-index-ramsize 4096
    +
    +

    All of the parameters in this command can be specified at the same time. To +change the username and password of the Couchbase Server administrator user, change +the port number to 5000, change the cluster name to "new_name", change the memory +quota of the data service to 2048MB and change the memory quota of the index +service to 4096MB run the following command.

    +
    +
    +
    $ couchbase-cli cluster-edit -c 192.168.1.5:8091 --username current_user \
+ --password current_password --cluster-username new_username \
+ --cluster-password new_password --cluster-port 5000 \
+ --cluster-name new_name --cluster-ramsize 2048 --cluster-index-ramsize 4096
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-cluster-init(1). +couchbase-cli-setting-cluster(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-cluster-init.dita b/content/cli/cbcli/couchbase-cli-cluster-init.dita deleted file mode 100644 index 4aa6e0d436..0000000000 --- a/content/cli/cbcli/couchbase-cli-cluster-init.dita +++ /dev/null @@ -1,252 +0,0 @@ - - - cluster-init - - Initializes a Couchbase Server cluster
    SYNTAX - - - couchbase-cli cluster-init [--cluster <url>] [--cluster-username <username>] - [--cluster-password <password>] [--cluster-port <port>] - [--cluster-ramsize <megabytes>] [--cluster-name <name>] - [--cluster-index-ramsize <megabytes>] [--cluster-fts-ramsize <megabytes>] - [--index-storage-setting <setting>] [--services <services>]
    DESCRIPTION - - -

    Initializes the first node in a Couchbase Server cluster. Before initializing a -cluster you need to decide which services you will be running on this node as -well as the memory quota for each service you are starting. Services started -with cluster-init will be running on this server only. Future servers that are -added to this cluster may be configured with different services -running on them. Note that the data service is always required on the first -node. Memory quotas for each service are global settings and will apply to each -node added to the cluster. The memory quota for a service only applies if that -service is running on a given server. You may also define the index storage mode -which will determine how the global secondary indexes (GSI) are stored.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of the cluster to initialize. By default, this - parameter is set to localhost:8091 for this command. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    ---cluster-username -
    -
    -

    - When starting a new cluster you need to create the Couchbase Server built-in - administrator user for the cluster. This user will be able to access the - Couchbase Server Administration Console as well as be used for data access - and future configuration. This option specifies the username for the - administrator user. -

    -
    -     - -
    ---cluster-password -
    -
    -

    - When starting a new cluster you need to create the Couchbase Server built-in - administrator user for the cluster. This user will be able to access the - Couchbase Server Administration Console as well as be used for data access - and future configuration. This option specifies the password for the - administrator user. -

    -
    -     - -
    ---cluster-ramsize -
    -
    -

    - Specifies the data services memory quota (in MB). This quota will be assigned - to all future nodes added to the cluster with the data service. -

    -
    -     - -
    ---cluster-fts-ramsize -
    -
    -

    - Sets the full-text service memory quota (in MB). This parameter is required - if the full-text service is specified in the --services option. This quota will - be assigned to all future nodes added to the cluster with the full-text - service. -

    -
    -     - -
    ---cluster-index-ramsize -
    -
    -

    - Sets the index service memory quota (in MB). This parameter is required if - the index service is specified in the --services option. This quota will be - assigned to all future nodes added to the cluster with the index service. -

    -
    -     - -
    ---cluster-name -
    -
    -

    - Sets the name for this cluster. Naming clusters is useful when you have - multiple Couchbase Server clusters in your deployment. -

    -
    -     - -
    ---cluster-port -
    -
    -

    - Specifies the port for the Couchbase Administration Console. Defaults to - port 8091. -

    -
    -     - -
    ---index-storage-setting -
    -
    -

    - Specifies the index storage mode for the index service. Accepted storage - modes are "default" for the standard index backend or "memopt" for memory - optimized indexes. If the index service is specified to be started with the - --services command then this parameter defaults to "default". If the index - service is not specified to be started then the index storage mode will not - be set. You will then be required to set the index storage mode when the - first index service is started on a server in the cluster. You may also - define the index storage mode even if an index service is not started on the - first node and it will be remembered when the first index service is added - in the future. You may not change this parameter while there are index nodes - in the cluster. -

    -
    -     - -
    ---services -
    -
    -

    - Specifies the services to start on this cluster. You may not change the - services running on this node once the cluster has been initialized. This - options takes a comma separated list of services. Accepted services are - "data", "index", "query", and "fts", specified as a comma-separated list. - This parameter defaults to "data". -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To create a Couchbase Server cluster with only the data service on the first node -and a memory quota of 4096MB run the following command.

    - 
    $ couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator \
- --cluster-password password --services data --cluster-ramsize 4096
    -

    To create a Couchbase Server cluster with the data and index service then you -also need to set the memory quotas for each service as well as the index storage -mode since you are starting the index service.

    -

    To create a cluster with an index memory quota of 1024MB, a data service memory -quota of 2048MB and a memory optimized index storage mode run the following -command.

    - 
    $ couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator \
- --cluster-password password --services data,index --cluster-ramsize 2048 \
- --cluster-index-ramsize 1024 --index-storage-setting memopt
    -

    To create a Couchbase Server cluster with all services then you need to set -the memory quotas for the data, index, and full-text service. The quotas are set -to 2048MB, 1024MB, and 1024MB respectively. A quota is not set -for the query service since it does not have a memory quota. You also need to set -the index storage mode for the index service, which will be set to "default" -since the service is started on the first node.

    - 
    $ couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator \
- --cluster-password password --services data,index,query,fts \
- --cluster-ramsize 2048 --cluster-index-ramsize 1024 \
- --cluster-fts-ramsize 1024 --index-storage-setting default \
    -

    If you want to set the port number you can do so with the --cluster-port -option. In the example below, a cluster is setup on port 5000 and starts -only the data service. The memory quota of the data service is set to 2048MB.

    - 
    $ couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator \
- --cluster-password password --services data --cluster-ramsize 2048 \
- --cluster-port 5000
    -
    DISCUSSION - - -

    The cluster-init subcommand sets up the first node on a Couchbase cluster. To -set per node settings such as the data storage directory, index storage -directory, or hostname see the -couchbase-cli-node-init1 - command. To -add nodes to a currently initialized cluster use the - -couchbase-cli-server-add1 - command. Some cluster settings may be -changed after a cluster is initialized. Use the - -couchbase-cli-setting-cluster1 - command to edit these settings.

    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-cluster-init.html b/content/cli/cbcli/couchbase-cli-cluster-init.html new file mode 100644 index 0000000000..759742f199 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-cluster-init.html @@ -0,0 +1,1031 @@ + + + + + +couchbase-cli-cluster-init(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli cluster-init [--cluster <url>] [--cluster-username <username>]
+          [--cluster-password <password>] [--cluster-port <port>]
+          [--cluster-ramsize <megabytes>] [--cluster-name <name>]
+          [--cluster-index-ramsize <megabytes>] [--cluster-fts-ramsize <megabytes>]
+          [--cluster-eventing-ramsize <megabytes>]
+          [--index-storage-setting <setting>] [--services <services>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Initializes the first node in a Couchbase Server cluster. Before initializing a +cluster you need to decide which services you will be running on this node as +well as the memory quota for each service you are starting. Services started +with cluster-init will be running on this server only. Future servers that are +added to this cluster may be configured with different services +running on them. Note that the data service is always required on the first +node. Memory quotas for each service are global settings and will apply to each +node added to the cluster. The memory quota for a service only applies if that +service is running on a given server. You may also define the index storage mode +which will determine how the global secondary indexes (GSI) are stored.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of the cluster to initialize. By default, this + parameter is set to localhost:8091 for this command. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +--cluster-username +
    +
    +

    + When starting a new cluster you need to create the Couchbase Server built-in + administrator user for the cluster. This user will be able to access the + Couchbase Server Administration Console as well as be used for data access + and future configuration. This option specifies the username for the + administrator user. +

    +
    +
    +--cluster-password +
    +
    +

    + When starting a new cluster you need to create the Couchbase Server built-in + administrator user for the cluster. This user will be able to access the + Couchbase Server Administration Console as well as be used for data access + and future configuration. This option specifies the password for the + administrator user. +

    +
    +
    +--cluster-ramsize +
    +
    +

    + Specifies the data services memory quota (in MB). This quota will be assigned + to all future nodes added to the cluster with the data service. +

    +
    +
    +--cluster-fts-ramsize +
    +
    +

    + Sets the full-text service memory quota (in MB). This parameter is required + if the full-text service is specified in the --services option. This quota will + be assigned to all future nodes added to the cluster with the full-text + service. +

    +
    +
    +--cluster-index-ramsize +
    +
    +

    + Sets the index service memory quota (in MB). This parameter is required if + the index service is specified in the --services option. This quota will be + assigned to all future nodes added to the cluster with the index service. +

    +
    +
    +--cluster-eventing-ramsize +
    +
    +

    + Sets the Eventing service memory quota (in MB). This parameter is required + if the Eventing service is specified in the --services option. This quota + will be assigned to all future nodes added to the cluster with the eventing + service. +

    +
    +
    +--cluster-name +
    +
    +

    + Sets the name for this cluster. Naming clusters is useful when you have + multiple Couchbase Server clusters in your deployment. +

    +
    +
    +--cluster-port +
    +
    +

    + Specifies the port for the Couchbase Administration Console. Defaults to + port 8091. +

    +
    +
    +--index-storage-setting +
    +
    +

    + Specifies the index storage mode for the index service. Accepted storage + modes are "default" for the standard index backend or "memopt" for memory + optimized indexes. If the index service is specified to be started with the + --services command then this parameter defaults to "default". If the index + service is not specified to be started then the index storage mode will not + be set. You will then be required to set the index storage mode when the + first index service is started on a server in the cluster. You may also + define the index storage mode even if an index service is not started on the + first node and it will be remembered when the first index service is added + in the future. You may not change this parameter while there are index nodes + in the cluster. +

    +
    +
    +--services +
    +
    +

    + Specifies the services to start on this cluster. You may not change the + services running on this node once the cluster has been initialized. This + options takes a comma separated list of services. Accepted services are + "data", "index", "query", and "fts", specified as a comma-separated list. + This parameter defaults to "data". +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To create a Couchbase Server cluster with only the data service on the first node +and a memory quota of 4096MB run the following command.

    +
    +
    +
    $ couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator \
+ --cluster-password password --services data --cluster-ramsize 4096
    +
    +

    To create a Couchbase Server cluster with the data and index service then you +also need to set the memory quotas for each service as well as the index storage +mode since you are starting the index service.

    +

    To create a cluster with an index memory quota of 1024MB, a data service memory +quota of 2048MB and a memory optimized index storage mode run the following +command.

    +
    +
    +
    $ couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator \
+ --cluster-password password --services data,index --cluster-ramsize 2048 \
+ --cluster-index-ramsize 1024 --index-storage-setting memopt
    +
    +

    To create a Couchbase Server cluster with all services then you need to set +the memory quotas for the Data, Index, Full-Text and Eventing service. The +quotas are set to 2048MB, 1024MB, 1024MB and 1024MB respectively. A quota is +not set for the query service since it does not have a memory quota. You also +need to set the index storage mode for the index service, which will be set to +"default" since the service is started on the first node.

    +
    +
    +
    $ couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator \
+ --cluster-password password --services data,index,query,fts \
+ --cluster-ramsize 2048 --cluster-index-ramsize 1024 \
+ --cluster-eventing-ramsize 1024\ --cluster-fts-ramsize 1024 \
+--index-storage-setting default
    +
    +

    If you want to set the port number you can do so with the --cluster-port +option. In the example below, a cluster is setup on port 5000 and starts +only the data service. The memory quota of the data service is set to 2048MB.

    +
    +
    +
    $ couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator \
+ --cluster-password password --services data --cluster-ramsize 2048 \
+ --cluster-port 5000
    +
    +
    +
    +
    +

    DISCUSSION

    +
    +

    The cluster-init subcommand sets up the first node on a Couchbase cluster. To +set per node settings such as the data storage directory, index storage +directory, or hostname see the couchbase-cli-node-init(1) command. To +add nodes to a currently initialized cluster use the +couchbase-cli-server-add(1) command. Some cluster settings may be +changed after a cluster is initialized. Use the +couchbase-cli-setting-cluster(1) command to edit these settings.

    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-cluster-edit(1). +couchbase-cli-node-init(1). +couchbase-cli-server-add(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the linkgit:couchbase-cli[1] suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-collect-logs-start.dita b/content/cli/cbcli/couchbase-cli-collect-logs-start.dita deleted file mode 100644 index d9426e066e..0000000000 --- a/content/cli/cbcli/couchbase-cli-collect-logs-start.dita +++ /dev/null @@ -1,206 +0,0 @@ - - - collect-logs-start - - Start log collection
    SYNTAX - - - couchbase-cli collect-logs-start [--cluster <url>] [--username <user>] - [--password <password>] [--all-nodes] [--nodes <node_list>] [--upload] - [--upload-host <host>] [--customer <customer>] [--ticket <ticket>]
    DESCRIPTION - - -

    This command collects all Couchbase Server log files from one or more nodes in -the cluster. Log collection is useful when there are failures in the cluster and -you need to figure out what is going on. Since there are many log files on -various different server the collect-logs-start command helps in aggregating all -of the different log file Couchbase Server creates.

    -

    To get the location of the collected log files you can run the - -collect-logs-status1 - command either while the log collection task -is running or after the log collection task has completed. If the --upload flag -is specified then the logs will also be uploaded to the host specified in the ---upload-host flag. The upload flag is intended for Couchbase Server -Enterprise Edition users who need to upload logs to the Couchbase Support Team -to aid in diagnosing support tickets that they have filed.

    -

    Note that only one log collection task may be running at any given time in the -cluster.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---all-nodes -
    -
    -

    - Specifies that log collection should take place on all nodes in the cluster. - May not be specified at the same time as --nodes. -

    -
    -     - -
    ---nodes <node_list> -
    -
    -

    - A list of one or more nodes to collect logs from, the nodes must include the - administrator port. The list should be comma - separated. This parameter may not be specified at the same time as the - --nodes option. -

    -
    -     - -
    ---upload -
    -
    -

    - Specifies that the logs should be uploaded to the the host specified with - the --upload-host option. This option should be used by Couchbase Server - Enterprise customers when uploading logs for the Couchbase Support Team. -

    -
    -     - -
    ---upload-host <host> -
    -
    -

    - Once log collection is completed, the logs zip file should be uploaded to - this URL. This parameter is required if the --upload flag is specified. -

    -
    -     - -
    ---customer <name> -
    -
    -

    - The name of the customer who is uploading these logs. This option is - required if the --upload flag is specified. -

    -
    -     - -
    ---ticket <num> -
    -
    -

    - The ticket number that the support team has created to track the issue - filed. This parameter is optional when specifying the --upload flag, but - recommended if you have a ticket number. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To collect logs on all nodes without uploading them run the following command.

    - 
    $ couchbase-cli collect-logs-start -c 192.168.1.5 --username Administrator \
- --password password --all-nodes
    -

    If there is a three node cluster with IP addresses 192.168.1.5, 192.168.1.6, and -192.168.1.7 and you only want to collect logs on the first two nodes and don’t -want to upload the logs then run the command below.

    - 
    $ couchbase-cli collect-logs-start -c 192.168.1.5 --username Administrator \
- --password password --nodes 192.168.1.5:8091,192.168.1.6:8091
    -

    If you are a Couchbase Server Enterprise Edition user and you need to upload logs -for all nodes for a support ticket then you can run the following command.

    - 
    $ couchbase-cli collect-logs-start -c 192.168.1.5 --username Administrator \
- --password password --all-nodes --upload --upload-customer customer_name \
- --upload-host s3.amazonaws.com/cb-customers --ticket 12345
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-collect-logs-start.html b/content/cli/cbcli/couchbase-cli-collect-logs-start.html new file mode 100644 index 0000000000..fb533cbd60 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-collect-logs-start.html @@ -0,0 +1,1033 @@ + + + + + +couchbase-cli-collect-logs-start(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli collect-logs-start [--cluster <url>] [--username <user>]
+          [--password <password>] [--all-nodes] [--nodes <node_list>]
+          [--redaction-level <level>] [--salt <string>]
+          [--output-directory <directory>] [--temporary-directory <directory>]
+          [--upload] [--upload-host <host>] [--upload-proxy <host>]
+          [--customer <customer>] [--ticket <ticket>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command collects all Couchbase Server log files from one or more nodes in +the cluster. Log collection is useful when there are failures in the cluster and +you need to figure out what is going on. Since there are many log files on +various different server the collect-logs-start command helps in aggregating all +of the different log file Couchbase Server creates.

    +

    To get the location of the collected log files you can run the +collect-logs-status(1) command either while the log collection task +is running or after the log collection task has completed. If the --upload flag +is specified then the logs will also be uploaded to the host specified in the +--upload-host flag. The upload flag is intended for Couchbase Server +Enterprise Edition users who need to upload logs to the Couchbase Support Team +to aid in diagnosing support tickets that they have filed.

    +

    Note that only one log collection task may be running at any given time in the +cluster.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--all-nodes +
    +
    +

    + Specifies that log collection should take place on all nodes in the cluster. + May not be specified at the same time as --nodes. +

    +
    +
    +--nodes <node_list> +
    +
    +

    + A list of one or more nodes to collect logs from, the nodes must include the + administrator port. The nodes should be comma separated. This option may + not be specified at the same time as the --all-nodes option. +

    +
    +
    +--redaction-level <level> +
    +
    +

    + Specifies what level the logs should be redacted too. This option has two + levels "none" and "partial". If partial is set then data such as + usernames, keys and values will be redacted from the logs. If none is set + then no data will be redacted. Please note that increasing the level + of redaction can have an impact on debugging problems. +

    +
    +
    +--salt <string> +
    +
    +

    + Specifies the string to used as the salt. This option can only be used + when --redaction-level is set to "partial". The salt is used to increase + the security of the redacted log files. If the option is not specified + then a salt is generated each time the logs are collected. It is important + to specify the salt if logs are from multiple collections or logs from client + SDKs have to be cross referenced against each other. +

    +
    +
    +--output-directory <directory> +
    +
    +

    + Specifies the directory on the node to place the logs. This is the location + where the logs will be placed after collection has finished. Log collection + will fail for the nodes that do not have this directory. It will also fail + if the directory does not have enough space. +

    +
    +
    +--temporary-directory <directory> +
    +
    +

    + Specifies the temporary directory on the node to use while generating the + logs. During log collection a number of temporary files are created before + being compressed. If this temporary location does not have enough space + log collection will fail. All nodes in the cluster must have this + directory. When it is not specified then the operating system temporary + directory will be used. +

    +
    +
    +--upload +
    +
    +

    + Specifies that the logs should be uploaded to the host specified with + the --upload-host option. This option should be used by Couchbase Server + Enterprise customers when uploading logs for the Couchbase Support Team. +

    +
    +
    +--upload-proxy <host> +
    +
    +

    + Specifies the proxy to use when uploading logs. This is useful when a + cluster is deployed in secure environments and do not have a direct + outbound connection to the internet to upload logs. +

    +
    +
    +--upload-host <host> +
    +
    +

    + Once log collection is completed, the logs zip file should be uploaded to + this URL. This parameter is required if the --upload flag is specified. +

    +
    +
    +--customer <name> +
    +
    +

    + The name of the customer who is uploading these logs. This option is + required if the --upload flag is specified. +

    +
    +
    +--ticket <num> +
    +
    +

    + The ticket number that the support team has created to track the issue + filed. This parameter is optional when specifying the --upload flag, but + recommended if you have a ticket number. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To collect logs on all nodes without uploading them run the following command.

    +
    +
    +
    $ couchbase-cli collect-logs-start -c 192.168.1.5 --username Administrator \
+ --password password --all-nodes
    +
    +

    If there is a three node cluster with IP addresses 192.168.1.5, 192.168.1.6, and +192.168.1.7 and you only want to collect logs on the first two nodes and don’t +want to upload the logs then run the command below.

    +
    +
    +
    $ couchbase-cli collect-logs-start -c 192.168.1.5 --username Administrator \
+ --password password --nodes 192.168.1.5:8091,192.168.1.6:8091
    +
    +

    If you are a Couchbase Server Enterprise Edition user and you need to upload logs +for all nodes for a support ticket then you can run the following command.

    +
    +
    +
    $ couchbase-cli collect-logs-start -c 192.168.1.5 --username Administrator \
+ --password password --all-nodes --upload --upload-customer customer_name \
+ --upload-host s3.amazonaws.com/cb-customers --ticket 12345
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-collect-logs-status(1). +couchbase-cli-collect-logs-stop(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-collect-logs-status.dita b/content/cli/cbcli/couchbase-cli-collect-logs-status.dita deleted file mode 100644 index 8fa6a3921a..0000000000 --- a/content/cli/cbcli/couchbase-cli-collect-logs-status.dita +++ /dev/null @@ -1,129 +0,0 @@ - - - collect-logs-status - - Get log collection status
    SYNTAX - - - couchbase-cli collect-logs-status [--cluster <url>] [--username <user>] - [--password <password>]
    DESCRIPTION - - -

    This command is used to check the status of the log collection task that is -either currently running or last completed. This command is especially useful -since it lists the exact location of the collected files on each node. Upon a -successful run of this command the output will either be one of the two messages -below.

    - 
    No log collection tasks were found
    - 
    or
    - 
    Status: <running | cancelled | completed>
-Details:
-        Node: <node name>
-        Status: <started | collected>
-        path: <path to the location of the collected logs>
    -

    If logs for multiple nodes were collected, then multiple nodes will be listed in -the details section.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To see the status of a running or completed log collection task run the command -below.

    - 
    $ couchbase-cli collect-logs-status -c 192.168.1.5 --username Administrator \
- --password password
    - 
    Status: completed
-Details:
-        Node: n_0@192.168.1.5
-        Status: collected
-        path: /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2016-09-27T005502-n_0@192.168.1.5.zip
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-collect-logs-status.html b/content/cli/cbcli/couchbase-cli-collect-logs-status.html new file mode 100644 index 0000000000..016ffbebae --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-collect-logs-status.html @@ -0,0 +1,915 @@ + + + + + +couchbase-cli-collect-logs-status(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli collect-logs-status [--cluster <url>] [--username <user>]
+          [--password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command is used to check the status of the log collection task that is +either currently running or last completed. This command is especially useful +since it lists the exact location of the collected files on each node. Upon a +successful run of this command the output will either be one of the two messages +below.

    +
    +
    +
    No log collection tasks were found
    +
    +
    +
    +
    or
    +
    +
    +
    +
    Status: <running | cancelled | completed>
+Details:
+        Node: <node name>
+        Status: <started | collected>
+        path: <path to the location of the collected logs>
    +
    +

    If logs for multiple nodes were collected, then multiple nodes will be listed in +the details section.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To see the status of a running or completed log collection task run the command +below.

    +
    +
    +
    $ couchbase-cli collect-logs-status -c 192.168.1.5 --username Administrator \
+ --password password
    +
    +
    +
    +
    Status: completed
+Details:
+        Node: n_0@192.168.1.5
+        Status: collected
+        path: /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2016-09-27T005502-n_0@192.168.1.5.zip
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-collect-logs-start(1). +couchbase-cli-collect-logs-stop(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-collect-logs-stop.dita b/content/cli/cbcli/couchbase-cli-collect-logs-stop.dita deleted file mode 100644 index 9a1c135ea5..0000000000 --- a/content/cli/cbcli/couchbase-cli-collect-logs-stop.dita +++ /dev/null @@ -1,111 +0,0 @@ - - - collect-logs-stop - - Stop the current log collection task
    SYNTAX - - - couchbase-cli collect-logs-stop [--cluster <url>] [--username <user>] - [--password <password>]
    DESCRIPTION - - -

    This command cancels the currently running log collection task.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To stop the currently running log collection task run the following command -below.

    - 
    $ couchbase-cli collect-logs-stop -c 192.168.1.5 --username Administrator \
- --password password
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-collect-logs-stop.html b/content/cli/cbcli/couchbase-cli-collect-logs-stop.html new file mode 100644 index 0000000000..e5cda9aadc --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-collect-logs-stop.html @@ -0,0 +1,885 @@ + + + + + +couchbase-cli-collect-logs-stop(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli collect-logs-stop [--cluster <url>] [--username <user>]
+          [--password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command cancels the currently running log collection task.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To stop the currently running log collection task run the following command +below.

    +
    +
    +
    $ couchbase-cli collect-logs-stop -c 192.168.1.5 --username Administrator \
+ --password password
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-collect-logs-start(1). +couchbase-cli-collect-logs-status(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-eventing-function-setup.html b/content/cli/cbcli/couchbase-cli-eventing-function-setup.html new file mode 100644 index 0000000000..c338b411df --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-eventing-function-setup.html @@ -0,0 +1,994 @@ + + + + + +couchbase-cli-eventing-function-setup(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli eventing-function-setup [--cluster <url>] [--username <user>]
+          [--password <password>]  [--import] [--export] [--export-all]
+          [--delete] [--list] [--deploy] [--undeploy] [--name <name>]
+          [--file <file>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command is used to manage functions in the Event service

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--import +
    +
    +

    + Import function(s) from a file. This option requires --file to be + specified. The file provided should be from the --export option. +

    +
    +
    +--export +
    +
    +

    + Export a named function to a file. This option requires --file and + --name to be specified. +

    +
    +
    +--export-all +
    +
    +

    + Export all functions to a file. This option requires --file to be + specified. +

    +
    +
    +--delete +
    +
    +

    + Deletes a named function. Functions can only be deleted when in the + un-deployed state. This option requires --name to be specified. +

    +
    +
    +--list +
    +
    +

    + List all functions. +

    +
    +
    +--deploy +
    +
    +

    + Deploys the named function. This option requires --name to be specified. +

    +
    +
    +--undeploy +
    +
    +

    + Undeploys the named function. This option requires --name to be specified. +

    +
    +
    +--name <name> +
    +
    +

    + The name of the function to take a action against. This is used by + --export, --delete, --deploy and --undeply options only. +

    +
    +
    +--file <file> +
    +
    +

    + The file to import and export functions to. This is used by --export and + --import options only. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To import functions from a file called functions.json:

    +
    +
    +
    $ couchbase-cli eventing-functions-setup -c 192.168.1.5 -u Administrator \
+ -p password --import --file functions.json
    +
    +

    To export a function called alert_function to a file called +functions.json:

    +
    +
    +
    $ couchbase-cli eventing-functions-setup -c 192.168.1.5 -u Administrator \
+ -p password --export --name alert_function --file functions.json
    +
    +

    To export all functions to a file called functions.json:

    +
    +
    +
    $ couchbase-cli eventing-functions-setup -c 192.168.1.5 -u Administrator \
+ -p password --export-all --file functions.json
    +
    +

    To list all functions:

    +
    +
    +
    $ couchbase-cli eventing-functions-setup -c 192.168.1.5 -u Administrator \
+ -p password --list
    +
    +

    To delete a function called alert_function:

    +
    +
    +
    $ couchbase-cli eventing-functions-setup -c 192.168.1.5 -u Administrator \
+ -p password --delete --name alert_function
    +
    +

    To deploy a function called alert_function:

    +
    +
    +
    $ couchbase-cli eventing-functions-setup -c 192.168.1.5 -u Administrator \
+ -p password --deploy --name alert_function
    +
    +

    To undeploy a function called alert_function:

    +
    +
    +
    $ couchbase-cli eventing-functions-setup -c 192.168.1.5 -u Administrator \
+ -p password --undeploy --name alert_function
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-failover.dita b/content/cli/cbcli/couchbase-cli-failover.dita deleted file mode 100644 index 3562d1d053..0000000000 --- a/content/cli/cbcli/couchbase-cli-failover.dita +++ /dev/null @@ -1,170 +0,0 @@ - - - failover - - Failover a node in the cluster
    SYNTAX - - - couchbase-cli failover [--cluster <url>] [--username <user>] - [--password <password>][--server-failover <server_list>] [--force] - [--no-progress-bar] [--no-wait]
    DESCRIPTION - - -

    This command fails over one or more nodes. Nodes can be either hard failed -over or gracefully failed over. A hard failover means that the failover happens -immediately, but risks potential data loss. Graceful failover ensures that -replication is up to date before the node is failed over so that there is no -data loss, but the failover is not immediate. Note that if a node is already down -you must perform a hard failover.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---server-failover <server_list> -
    -
    -

    - A comma separated list of nodes to failover. -

    -
    -     - -
    ---force -
    -
    -

    - Specifying this flag signifies that the nodes to be failed over should be - hard failed over. A hard failover means that the failover is immediate, but - you risk potential data loss. If this flag is not specified then the - failover will occur only once replication is up to date. This means there - will be no data loss, but the failover is not immediate. Note that if a - server is already down you must hard fail it over. -

    -
    -     - -
    ---no-progress-bar -
    -
    -

    - Disables showing the progress bar which tracks the progress of the - rebalance. Note that a rebalance only occurs for graceful failovers (e.g. - non-forced failovers). The failover command will still wait for rebalance - completion even if this flag is specified, but the progress bar will not - be shown. -

    -
    -     - -
    ---no-wait -
    -
    -

    - Specifies that this command should not wait for the completion of rebalance - before exiting. Note that a rebalance only occurs for graceful failovers - (eg. non-forced failovers). -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To hard fail over a node run the following command:

    - 
    $ couchbase-cli failover -c 192.168.1.5:8091 --username Administrator \
- --password password --server-failover 192.168.1.6:8091 --force
    -

    To gracefully failover a node run the following command:

    - 
    $ couchbase-cli failover -c 192.168.1.5:8091 --username Administrator \
- --password password --server-failover 192.168.1.6:8091
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-failover.html b/content/cli/cbcli/couchbase-cli-failover.html new file mode 100644 index 0000000000..7ccff38278 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-failover.html @@ -0,0 +1,942 @@ + + + + + +couchbase-cli-failover(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli failover [--cluster <url>] [--username <user>]
+          [--password <password>][--server-failover <server_list>] [--force]
+          [--no-progress-bar] [--no-wait]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command fails over one or more nodes. Nodes can be either hard failed +over or gracefully failed over. A hard failover means that the failover happens +immediately, but risks potential data loss. Graceful failover ensures that +replication is up to date before the node is failed over so that there is no +data loss, but the failover is not immediate. Note that if a node is already down +you must perform a hard failover.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--server-failover <server_list> +
    +
    +

    + A comma separated list of nodes to failover. +

    +
    +
    +--force +
    +
    +

    + Specifying this flag signifies that the nodes to be failed over should be + hard failed over. A hard failover means that the failover is immediate, but + you risk potential data loss. If this flag is not specified then the + failover will occur only once replication is up to date. This means there + will be no data loss, but the failover is not immediate. Note that if a + server is already down you must hard fail it over. +

    +
    +
    +--no-progress-bar +
    +
    +

    + Disables showing the progress bar which tracks the progress of the + rebalance. Note that a rebalance only occurs for graceful failovers (e.g. + non-forced failovers). The failover command will still wait for rebalance + completion even if this flag is specified, but the progress bar will not + be shown. +

    +
    +
    +--no-wait +
    +
    +

    + Specifies that this command should not wait for the completion of rebalance + before exiting. Note that a rebalance only occurs for graceful failovers + (eg. non-forced failovers). +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To hard fail over two nodes run the following command:

    +
    +
    +
    $ couchbase-cli failover -c 192.168.1.5:8091 --username Administrator \
+ --password password \
+ --server-failover 192.168.1.6:8091,192.168.1.7:8091 --force
    +
    +

    To gracefully failover a node run the following command:

    +
    +
    +
    $ couchbase-cli failover -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-failover 192.168.1.6:8091
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-rebalance(1). +couchbase-cli-recovery(1). +couchbase-cli-server-add(1). +couchbase-cli-setting-autofailover(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-group-manage.dita b/content/cli/cbcli/couchbase-cli-group-manage.dita deleted file mode 100644 index 2a25758f9e..0000000000 --- a/content/cli/cbcli/couchbase-cli-group-manage.dita +++ /dev/null @@ -1,211 +0,0 @@ - - - group-manage - - Manage server groups
    SYNTAX - - - couchbase-cli group-manage [--cluster <url>] [--username <user>] - [--password <password>] [--create] [--delete] [--list] [--rename <group>] - [--move-servers] [--group-name <name>] [--from-group <name>] - [--to-group <name>]
    DESCRIPTION - - -

    This command is used to manage server groups in a Couchbase cluster. When moving -servers between groups it is possible that you may need to rebalance the cluster -since the cluster topology may be changing. You should always run - -couchbase-cli-rebalance1 - after running this command.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---create -
    -
    -

    - Creates a new server group. -

    -
    -     - -
    ---delete -
    -
    -

    - Deletes a server group. -

    -
    -     - -
    ---list -
    -
    -

    - Lists all server groups. -

    -
    -     - -
    ---rename <group> -
    -
    -

    - Renames a server group. -

    -
    -     - -
    ---move-servers <server_list> -
    -
    -

    - A list of servers to move between groups. -

    -
    -     - -
    ---group-name <name> -
    -
    -

    - The name of the server group. -

    -
    -     - -
    ---from-group <name> -
    -
    -

    - The group to move servers from. -

    -
    -     - -
    ---to-group <name> -
    -
    -

    - The group to move servers to. This group must already exists. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To create a new server group called "rack1" run the following command.

    - 
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
- -p password --create --group-name rack1
    -

    To rename the server group "rack1" to "rack2" run the following command.

    - 
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
- -p password --rename rack1 --group-name rack2
    -

    To list all of the server groups in a cluster run the following command.

    - 
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
- -p password --list
    -

    To delete the server group "rack2" run the following command.

    - 
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
- -p password --delete --group-name rack2
    -

    To move the servers 192.168.1.5 and 192.168.1.6 from server group "rack3" to -server group "rack4" run the following command.

    - 
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
- -p password --move-servers 192.168.1.5,192.168.1.6 --from-group rack3 \
- --to-group rack4
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-group-manage.html b/content/cli/cbcli/couchbase-cli-group-manage.html new file mode 100644 index 0000000000..a8b0efbf9a --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-group-manage.html @@ -0,0 +1,979 @@ + + + + + +couchbase-cli-group-manage(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli group-manage [--cluster <url>] [--username <user>]
+          [--password <password>] [--create] [--delete] [--list] [--rename <group>]
+          [--move-servers] [--group-name <name>] [--from-group <name>]
+          [--to-group <name>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command is used to manage server groups in a Couchbase cluster. When moving +servers between groups it is possible that you may need to rebalance the cluster +since the cluster topology may be changing. You should always run +couchbase-cli-rebalance(1) after running this command.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--create +
    +
    +

    + Creates a new server group. +

    +
    +
    +--delete +
    +
    +

    + Deletes a server group. +

    +
    +
    +--list +
    +
    +

    + Lists all server groups. +

    +
    +
    +--rename <group> +
    +
    +

    + Renames a server group. +

    +
    +
    +--move-servers <server_list> +
    +
    +

    + A list of servers to move between groups. +

    +
    +
    +--group-name <name> +
    +
    +

    + The name of the server group. +

    +
    +
    +--from-group <name> +
    +
    +

    + The group to move servers from. +

    +
    +
    +--to-group <name> +
    +
    +

    + The group to move servers to. This group must already exists. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To create a new server group called "rack1" run the following command.

    +
    +
    +
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
+ -p password --create --group-name rack1
    +
    +

    To rename the server group "rack1" to "rack2" run the following command.

    +
    +
    +
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
+ -p password --rename rack1 --group-name rack2
    +
    +

    To list all of the server groups in a cluster run the following command.

    +
    +
    +
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
+ -p password --list
    +
    +

    To delete the server group "rack2" run the following command.

    +
    +
    +
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
+ -p password --delete --group-name rack2
    +
    +

    To move the servers 192.168.1.5 and 192.168.1.6 from server group "rack3" to +server group "rack4" run the following command.

    +
    +
    +
    $ couchbase-cli group-manage -c 192.168.1.5 -u Administrator \
+ -p password --move-servers 192.168.1.5,192.168.1.6 --from-group rack3 \
+ --to-group rack4
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-rebalance(1) +couchbase-cli-server-add(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-help.html b/content/cli/cbcli/couchbase-cli-help.html new file mode 100644 index 0000000000..0e65907651 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-help.html @@ -0,0 +1,883 @@ + + + + + +couchbase-cli-help(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli help
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This page is not yet completed.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    TBD

    +
    +
    +
    +

    DISCUSSION

    +
    +

    TBD

    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    TBD

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-host-list.dita b/content/cli/cbcli/couchbase-cli-host-list.dita deleted file mode 100644 index ee6080c5c4..0000000000 --- a/content/cli/cbcli/couchbase-cli-host-list.dita +++ /dev/null @@ -1,117 +0,0 @@ - - - host-list - - Lists all hosts in the cluster
    SYNTAX - - - couchbase-cli host-list [--cluster <url>] [--username <user>] - [--password <password>]
    DESCRIPTION - - -

    Displays a list of hostnames corresponding to each node in the cluster. Each -hostname is printed to stdout on its own line. The hostname may or may not -contain a port number. If the port number is not a part of the hostname, then -the port number is assumed to be 8091.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To get a list of hostnames for each node in the cluster run the following -command:

    - 
    $ couchbase-cli host-list -c 192.168.1.5:8091 --username Administrator \
- --password password
    - 
    cb1.mydomain.com:8091
-cb2.mydomain.com:8091
-cb3.mydomain.com:8091
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-host-list.html b/content/cli/cbcli/couchbase-cli-host-list.html new file mode 100644 index 0000000000..17c2b9c604 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-host-list.html @@ -0,0 +1,894 @@ + + + + + +couchbase-cli-host-list(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli host-list [--cluster <url>] [--username <user>]
+          [--password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Displays a list of hostnames corresponding to each node in the cluster. Each +hostname is printed to stdout on its own line. The hostname may or may not +contain a port number. If the port number is not a part of the hostname, then +the port number is assumed to be 8091.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To get a list of hostnames for each node in the cluster run the following +command:

    +
    +
    +
    $ couchbase-cli host-list -c 192.168.1.5:8091 --username Administrator \
+ --password password
    +
    +
    +
    +
    cb1.mydomain.com:8091
+cb2.mydomain.com:8091
+cb3.mydomain.com:8091
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-server-info(1) +couchbase-cli-server-list(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-master-password.dita b/content/cli/cbcli/couchbase-cli-master-password.dita deleted file mode 100644 index 039e04b99b..0000000000 --- a/content/cli/cbcli/couchbase-cli-master-password.dita +++ /dev/null @@ -1,185 +0,0 @@ - - - master-password - - Manages and sends the Couchbase master password
    SYNTAX - - - couchbase-cli master-password [--cluster <url>] [--username <user>] - [--password <password>] [--send-password <password>] - [--new-password <password>] [--rotate-password]
    DESCRIPTION - - -

    Couchbase Server Enterprise Edition has a "Secret Management" feature which -allows users to securely encrypt passwords and other sensitive configuration -information that is stored on disk. These secrets must be stored in a secure way -and access must be controlled to reduce the risk of accidental exposure. By -using Secret Management in Couchbase Server, secrets are written to disk in -encrypted format. To decrypt these secrets Couchbase requires entering a -"master password" which is supplied by the user during server startup. This -master password can be passed to the server using the couchbase-cli -master-password command.

    -

    By default the Secret Management feature is disabled. To enable the feature you -must first set the master password. Once a master password is set the user is -required to enter it when the server starts up. This can be done by either -setting the environment variable CB_MASTER_PASSWORD=<password> during server -startup. Alternatively, you can set the environment variable -CB_WAIT_FOR_MASTER_PASSWORD=true and then enter the master password using the -couchbase-cli master-password command. See the EXAMPLES section for more details -about how to use the Secret Management feature.

    -

    Note: The master password subcommand must be run on the same node as the -server it is targeting because it requires access to files in the installation -in order to

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---rotate-data-key -
    -
    -

    - Secrets are encrypted using a data key file which is a unique key that is - stored on disk for each server. To open this file the master password is - used to generate a key which decrypts the contents of the data key file. - The contents of the decrypted data key file can then be used to decrypt - secrets. Some users may want to generate a new data key file periodically - to increase security. This option is used to generate a new data key file. -

    -
    -     - -
    ---new-password -
    -
    -

    - Sets a new master password for the server specified. The user may specify - this password on the command line or through non-echoed stdin. To specify - the password through non-echoed stdin do not provide a value for this - option. The user will then be prompted to enter the password. -

    -
    -     - -
    ---send-password -
    -
    -

    - Sends the master password to server which is waiting to start up. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To use the Secret Management feature the first thing you need to do is set a -password on each node of the cluster. To do this install and start Couchbase, -but don’t go through the setup process or initialize the cluster. Once Couchbase -has started run the following command to set the master password for your -server. Note that you must be running the command from the same machine as the -server you are connecting to.

    - 
    $ couchbase-cli master-password -c 127.0.0.1 -u Administrator -p password \
-  --new-password password
    -

    Now that the master password has been set you need to set the server environment -variable CB_WAIT_FOR_MASTER_PASSWORD=true. You can do this by running the -command below or by setting the variable in your .bashrc file.

    - 
    $ export CB_WAIT_FOR_MASTER_PASSWORD=true
    -

    This environment variable will cause Couchbase to wait for the master password -before starting up. Now that it is set you need to restart your cluster. Upon -restarting the cluster you will notice that the server doesn’t fully start. This -is because it is waiting for you to enter the master password. You can do this -by running the command below. This command must also be run on the same machine -that the server you ar starting is on.

    - 
    $ couchbase-cli master-password -c 127.0.0.1 -u Administrator -p password \
-  --send-password password
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-master-password.html b/content/cli/cbcli/couchbase-cli-master-password.html new file mode 100644 index 0000000000..39bf7f8eaf --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-master-password.html @@ -0,0 +1,928 @@ + + + + + +couchbase-cli-master-password(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli master-password [--new-password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Couchbase Server Enterprise Edition has a "Secret Management" feature, which +allows users to securely encrypt passwords and other sensitive configuration +information that is stored on disk. These secrets must be stored in a secure +way, and access must be controlled to reduce the risk of accidental exposure. +By using Secret Management in Couchbase Server, secrets are written to disk +in encrypted format. To decrypt these secrets, Couchbase requires the +entering of a "master password", which is supplied by the user during server +startup. This master password can be passed to the server using this command.

    +

    By default the Secret Management feature is disabled. To enable the feature, +you must first set the master password. Once a master password is set, the +user is required to enter it when the server starts up. This can be done by +setting the environment variable CB_MASTER_PASSWORD=<password> during server +startup. Alternatively, you can set the environment variable +CB_WAIT_FOR_MASTER_PASSWORD=true, and then enter the master password using the +couchbase-cli master-password command. This command must be run locally on the +node that needs to be unlocked.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--send-password +
    +
    +

    + Sends the master password to the server that is waiting to start up. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To use the Secret Management feature, the first thing you need to do is set a +password on each node of the cluster. To do this, install and start Couchbase, +but don’t go through the setup process or initialize the cluster. Once +Couchbase has started, run the following command to set the master password +for your server.

    +
    +
    +
    $ couchbase-cli setting-master-password -c 127.0.0.1 -u Administrator \
+  -p password --new-password password
    +
    +

    Once the master password has been set, you need to set the server environment +variable CB_WAIT_FOR_MASTER_PASSWORD=true. You can do this by running the +command below or by setting the variable in your .bashrc file.

    +
    +
    +
    $ export CB_WAIT_FOR_MASTER_PASSWORD=true
    +
    +

    This environment variable will cause Couchbase to wait for the master password +before starting up. Once it is set, you need to restart your cluster. Upon +restarting the cluster you will notice that the server doesn’t fully start. +This is because it is waiting for you to enter the master password. You can do +this by running the command below. The master-passowrd subcommand has to be +run locally on the node that is waiting for the master password.

    +
    +
    +
    $ couchbase-cli master-password --send-password password
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-master-password(1). +couchbase-cli-cluster-init(1). +couchbase-cli-server-add(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-node-init.dita b/content/cli/cbcli/couchbase-cli-node-init.dita deleted file mode 100644 index f8b5988b08..0000000000 --- a/content/cli/cbcli/couchbase-cli-node-init.dita +++ /dev/null @@ -1,165 +0,0 @@ - - - node-init - - Initializes a node
    SYNTAX - - - couchbase-cli node-init [--cluster <url>] [--username <user>] - [--password <password>] [--node-init-data-path <path>] - [--node-init-index-path <path>] [--node-init-hostname <hostname>]
    DESCRIPTION - - -

    This command initializes a Couchbase Server node. In particular this command -allows the user to set the node’s data path, index path, and hostname. The data -path, index path and hostname must be set prior to initializing the cluster or -adding the node to an existing cluster. The hostname however can be -changed later if the node is the only node in the cluster.

    -

    To get the best performance from Couchbase Server, it is recommended that the -data and index paths be set to separate disks.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---node-init-data-path -
    -
    -

    - The path to store data files create by the Couchbase data service. Note that - this path is also where view indexes are written on this server. This flag - can only be specified against a node that is not yet part of a cluster. -

    -
    -     - -
    ---node-init-index-path -
    -
    -

    - The path to store files create by the Couchbase index service. This flag can - only be specified against a node that is not yet part of a cluster. -

    -
    -     - -
    ---node-init-hostname -
    -
    -

    - Specifies the hostname for this server. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To initialize a node and set the index path to /mnt1/indexes and the data path -to /mnt2/data run the following command. Note that this command must be run -before the node becomes part of a cluster.

    - 
    $ couchbase-cli node-init -c 192.168.1.5 \
- --node-init-data-path /mnt2/data --node-init-index-path /mnt1/indexes
    -

    In the command above, the cluster username and password have not yet been set so -you can skip adding these to the command line. If you later initialize the cluster -and want to set the hostname for the cluster, then run the command -below. Note that this time the cluster is initialized, so you must include the -username and password.

    - 
    $ couchbase-cli node-init -c 192.168.1.5 -u Administrator -p password \
- --hostname cb1.mydomain.com
    -

    You could have alternatively set all three of these options below before -initializing the cluster. This can be done with the following command.

    - 
    $ couchbase-cli node-init -c 192.168.1.5 --hostname cb1.mydomain.com \
- --node-init-data-path /mnt2/data --node-init-index-path /mnt1/indexes
    -

    Again, note that you don’t need the username and password because in this example -the cluster has not yet been initialized.

    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-node-init.html b/content/cli/cbcli/couchbase-cli-node-init.html new file mode 100644 index 0000000000..1ceadef4ea --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-node-init.html @@ -0,0 +1,939 @@ + + + + + +couchbase-cli-node-init(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli node-init [--cluster <url>] [--username <user>]
+          [--password <password>] [--node-init-data-path <path>]
+          [--node-init-index-path <path>] [--node-init-hostname <hostname>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command initializes a Couchbase Server node. In particular this command +allows the user to set the node’s data path, index path, and hostname. The data +path, index path and hostname must be set prior to initializing the cluster or +adding the node to an existing cluster. The hostname however can be +changed later if the node is the only node in the cluster.

    +

    To get the best performance from Couchbase Server, it is recommended that the +data and index paths be set to separate disks.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--node-init-data-path +
    +
    +

    + The path to store data files create by the Couchbase data service. Note that + this path is also where view indexes are written on this server. This flag + can only be specified against a node that is not yet part of a cluster. +

    +
    +
    +--node-init-index-path +
    +
    +

    + The path to store files create by the Couchbase index service. This flag can + only be specified against a node that is not yet part of a cluster. +

    +
    +
    +--node-init-hostname +
    +
    +

    + Specifies the hostname for this server. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To initialize a node and set the index path to /mnt1/indexes and the data path +to /mnt2/data run the following command. Note that this command must be run +before the node becomes part of a cluster.

    +
    +
    +
    $ couchbase-cli node-init -c 192.168.1.5 \
+ --node-init-data-path /mnt2/data --node-init-index-path /mnt1/indexes
    +
    +

    In the command above, the cluster username and password have not yet been set so +you can skip adding these to the command line. If you later initialize the cluster +and want to set the hostname for the cluster, then run the command +below. Note that this time the cluster is initialized, so you must include the +username and password.

    +
    +
    +
    $ couchbase-cli node-init -c 192.168.1.5 -u Administrator -p password \
+ --hostname cb1.mydomain.com
    +
    +

    You could have alternatively set all three of these options below before +initializing the cluster. This can be done with the following command.

    +
    +
    +
    $ couchbase-cli node-init -c 192.168.1.5 --hostname cb1.mydomain.com \
+ --node-init-data-path /mnt2/data --node-init-index-path /mnt1/indexes
    +
    +

    Again, note that you don’t need the username and password because in this example +the cluster has not yet been initialized.

    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-cluster-init(1). +couchbase-cli-server-add(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-rebalance-status.dita b/content/cli/cbcli/couchbase-cli-rebalance-status.dita deleted file mode 100644 index e2f17b3cf8..0000000000 --- a/content/cli/cbcli/couchbase-cli-rebalance-status.dita +++ /dev/null @@ -1,111 +0,0 @@ - - - rebalance-status - - Show the current rebalance status
    SYNTAX - - - couchbase-cli rebalance-status [--cluster <url>] [--username <user>] - [--password <password>]
    DESCRIPTION - - -

    Gets rebalance status information for the cluster. Currently the status either -reports "running" or "not running".

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To get the current rebalance status run the following command.

    - 
    $ couchbase-cli rebalance-status -c 192.168.1.5 --username Administrator \
- --password password
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-rebalance-status.html b/content/cli/cbcli/couchbase-cli-rebalance-status.html new file mode 100644 index 0000000000..10f9f6b0cb --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-rebalance-status.html @@ -0,0 +1,885 @@ + + + + + +couchbase-cli-rebalance-status(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli rebalance-status [--cluster <url>] [--username <user>]
+          [--password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Gets rebalance status information for the cluster. Currently the status either +reports "running" or "not running".

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To get the current rebalance status run the following command.

    +
    +
    +
    $ couchbase-cli rebalance-status -c 192.168.1.5 --username Administrator \
+ --password password
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-rebalance(1). +couchbase-cli-rebalance-stop(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-rebalance-stop.dita b/content/cli/cbcli/couchbase-cli-rebalance-stop.dita deleted file mode 100644 index e9dc37d37c..0000000000 --- a/content/cli/cbcli/couchbase-cli-rebalance-stop.dita +++ /dev/null @@ -1,110 +0,0 @@ - - - rebalance-stop - - Stops the current rebalance task
    SYNTAX - - - couchbase-cli rebalance-stop [--cluster <url>] [--username <user>] - [--password <password>]
    DESCRIPTION - - -

    Stops the currently running rebalance.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To stop the currently running rebalance, run the following command.

    - 
    $ couchbase-cli rebalance-stop -c 192.168.1.5 --username Administrator \
- --password password
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-rebalance-stop.html b/content/cli/cbcli/couchbase-cli-rebalance-stop.html new file mode 100644 index 0000000000..04c019ee41 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-rebalance-stop.html @@ -0,0 +1,884 @@ + + + + + +couchbase-cli-rebalance-stop(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli rebalance-stop [--cluster <url>] [--username <user>]
+          [--password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Stops the currently running rebalance.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To stop the currently running rebalance, run the following command.

    +
    +
    +
    $ couchbase-cli rebalance-stop -c 192.168.1.5 --username Administrator \
+ --password password
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-rebalance(1). +couchbase-cli-rebalance-status(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-rebalance.dita b/content/cli/cbcli/couchbase-cli-rebalance.dita deleted file mode 100644 index 3a116a53c0..0000000000 --- a/content/cli/cbcli/couchbase-cli-rebalance.dita +++ /dev/null @@ -1,195 +0,0 @@ - - - rebalance - - Rebalances data and indexes across nodes in a cluster
    SYNTAX - - - couchbase-cli rebalance [--cluster <url>] [--username <user>] - [--password <password>] [--server-remove <servers>] [--no-progress-bar] - [--no-wait]
    DESCRIPTION - - -

    Rebalances data and indexes across all nodes in the cluster. This command -should be used after nodes are added, removed, or failed over from the cluster -in order to ensure that each node in the cluster has a similar "balanced" amount -of data and indexes.

    -

    To add nodes use the -couchbase-cli-server-add1 - subcommand. To -remove nodes specify the list of nodes to remove using the --server-remove -option in the rebalance subcommand. To failover nodes see the - -couchbase-cli-failover1 - subcommand. After running the - -couchbase-cli-server-add1 - or -couchbase-cli-failover1 - -subcommands ensure that you run the rebalance command to balance data and indexes -across the cluster.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---server-remove <servers> -
    -
    -

    - A comma separated list of nodes to remove from the cluster. Each node - in the list should correspond to the hostname or IP address of that - server. -

    -
    -     - -
    ---no-progress-bar -
    -
    -

    - Hides the progress bar which tracks the progress of the - rebalance. This command will still wait for rebalance completion even if - this flag is specified, but the progress bar will not be shown. -

    -
    -     - -
    ---no-wait -
    -
    -

    - Specifies that this command should not wait for the completion of rebalance - before exiting. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To rebalance a new node into the cluster you would first add a node using -the -couchbase-cli-server-add1 - command and then start the rebalance -with the rebalance command.

    - 
    $ couchbase-cli server-add -c 192.168.1.5:8091 --username Administrator \
- --password password --server-add 192.168.1.6:8091 \
- --server-add-username Administrator --server-add-password password
    - 
    $ couchbase-cli rebalance -c 192.168.1.5:8091 --username Administrator \
- --password password
    -

    If you just want to remove a node from the cluster and start a rebalance, -just run the rebalance command.

    - 
    $ couchbase-cli rebalance -c 192.168.1.5:8091 --username Administrator \
- --password password --server-remove 192.168.1.6:8091
    -

    You can remove multiple nodes at once and run the rebalance with the -command below. Note that it will be faster overall removing multiple nodes -at once as opposed to removing them one at a time.

    - 
    $ couchbase-cli rebalance -c 192.168.1.5:8091 --username Administrator \
- --password password --server-remove 192.168.1.6:8091,192.168.1.7:8091
    -

    To add a node and remove another node you need to run the - -couchbase-cli-server-add1 - command before starting the rebalance, -as shown below.

    - 
    $ couchbase-cli server-add -c 192.168.1.5:8091 --username Administrator \
- --password password --server-add 192.168.1.6:8091 \
- --server-add-username Administrator --server-add-password password
    - 
    $ couchbase-cli rebalance -c 192.168.1.5:8091 --username Administrator \
- --password password --server-remove 192.168.1.7:8091
    -

    If you add one node to the cluster and remove a node during the same -rebalance, Couchbase Server will do a "swap rebalance". This means data and -indexes from the node being removed are moved to the one being added. This -means the rebalance will only occur between these two nodes as -opposed to involving all nodes in the cluster.

    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-rebalance.html b/content/cli/cbcli/couchbase-cli-rebalance.html new file mode 100644 index 0000000000..ac054c2959 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-rebalance.html @@ -0,0 +1,967 @@ + + + + + +couchbase-cli-rebalance(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli rebalance [--cluster <url>] [--username <user>]
+          [--password <password>] [--server-remove <servers>] [--no-progress-bar]
+          [--no-wait]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Rebalances data and indexes across all nodes in the cluster. This command +should be used after nodes are added, removed, or failed over from the cluster +in order to ensure that each node in the cluster has a similar "balanced" amount +of data and indexes.

    +

    To add nodes use the couchbase-cli-server-add(1) subcommand. To +remove nodes specify the list of nodes to remove using the --server-remove +option in the rebalance subcommand. To failover nodes see the +couchbase-cli-failover(1) subcommand. After running the +couchbase-cli-server-add(1) or couchbase-cli-failover(1) +subcommands ensure that you run the rebalance command to balance data and indexes +across the cluster.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--server-remove <servers> +
    +
    +

    + A comma separated list of nodes to remove from the cluster. Each node + in the list should correspond to the hostname or IP address of that + server. +

    +
    +
    +--no-progress-bar +
    +
    +

    + Hides the progress bar which tracks the progress of the + rebalance. This command will still wait for rebalance completion even if + this flag is specified, but the progress bar will not be shown. +

    +
    +
    +--no-wait +
    +
    +

    + Specifies that this command should not wait for the completion of rebalance + before exiting. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To rebalance a new node into the cluster you would first add a node using +the couchbase-cli-server-add(1) command and then start the rebalance +with the rebalance command.

    +
    +
    +
    $ couchbase-cli server-add -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-add 192.168.1.6:8091 \
+ --server-add-username Administrator --server-add-password password
    +
    +
    +
    +
    $ couchbase-cli rebalance -c 192.168.1.5:8091 --username Administrator \
+ --password password
    +
    +

    If you just want to remove a node from the cluster and start a rebalance, +just run the rebalance command.

    +
    +
    +
    $ couchbase-cli rebalance -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-remove 192.168.1.6:8091
    +
    +

    You can remove multiple nodes at once and run the rebalance with the +command below. Note that it will be faster overall removing multiple nodes +at once as opposed to removing them one at a time.

    +
    +
    +
    $ couchbase-cli rebalance -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-remove 192.168.1.6:8091,192.168.1.7:8091
    +
    +

    To add a node and remove another node you need to run the +couchbase-cli-server-add(1) command before starting the rebalance, +as shown below.

    +
    +
    +
    $ couchbase-cli server-add -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-add 192.168.1.6:8091 \
+ --server-add-username Administrator --server-add-password password
    +
    +
    +
    +
    $ couchbase-cli rebalance -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-remove 192.168.1.7:8091
    +
    +

    If you add one node to the cluster and remove a node during the same +rebalance, Couchbase Server will do a "swap rebalance". This means data and +indexes from the node being removed are moved to the one being added. This +means the rebalance will only occur between these two nodes as +opposed to involving all nodes in the cluster.

    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-failover(1). +couchbase-cli-recovery(1). +couchbase-cli-server-add(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-recovery.dita b/content/cli/cbcli/couchbase-cli-recovery.dita deleted file mode 100644 index 375c8232cd..0000000000 --- a/content/cli/cbcli/couchbase-cli-recovery.dita +++ /dev/null @@ -1,166 +0,0 @@ - - - recovery - - Recovers a previously failed over node
    SYNTAX - - - couchbase-cli recovery [--cluster <url>] [--username <username>] - [--password <password>] [--server-recovery <servers>] - [--recovery-type <type>]
    DESCRIPTION - - -

    When a server is failed over and removed from the cluster it may only be able to -be added back to the cluster. And example of this is when a server loses power. -This server might get failed over and removed from the cluster, but once power -is restored to the server you may want to add it back to the cluster.

    -

    Another use case is taking a server out of the cluster for maintainance. This -is done by gracefully failing over a server to ensure there is no data loss. -The administrator can then perform mainainance on the removed server and add it -back with the server-readd command.

    -

    The recovery command also allows a server to have it’s data remvoed before being -added back to the cluster (full recovery) or having the server resume from where -it last left off (delta recovery). Delta recovery will always take the least -amount of time and is the recommended recovery mode.

    -

    Note that after the recovery subcommand is run you must rebalance the -cluster. See the -couchbase-rebalance1 - command for more information -on rebalancing a cluster.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---server-recovery <servers> -
    -
    -

    - A command separated list of servers to recover. The each server should - correspond to the hostname or IP address of a server to be added back to - the cluster. -

    -
    -     - -
    ---recovery-type <type> -
    -
    -

    - Specifies whether or not we should do full or delta recovery of data on the - servers being added back. Full recover means that all data on the servers - being recovered will be removed and we will recover all data. Specifying - delta recover means we will keep the data that is on each server and only - recover the data that each server doesn’t have. To specify full recovery set - this option to "full". For delta recovery set this option to "delta". The - default value for this option is "full". -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    If we have a server at 192.168.1.6:8091 that we want to recover and we want to -recover data using delta recovery then we can run the following command.

    - 
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
- --password password --server-recovery 192.168.1.6:8091 \
- --recovery-type delta
    -

    If we need to recover multiple servers then we would run the following command.

    - 
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
- --password password --server-recovery 192.168.1.6:8091,192.168.1.7:8091 \
- --recovery-type delta
    -

    If we wanted the servers being recovered to use full recovery then we would run -the following command.

    - 
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
- --password password --server-recovery 192.168.1.6:8091 \
- --recovery-type full
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-recovery.html b/content/cli/cbcli/couchbase-cli-recovery.html new file mode 100644 index 0000000000..000aa15206 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-recovery.html @@ -0,0 +1,940 @@ + + + + + +couchbase-cli-recovery(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli recovery [--cluster <url>] [--username <username>]
+          [--password <password>] [--server-recovery <servers>]
+          [--recovery-type <type>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    When a server is failed over and removed from the cluster it may only be able to +be added back to the cluster. And example of this is when a server loses power. +This server might get failed over and removed from the cluster, but once power +is restored to the server you may want to add it back to the cluster.

    +

    Another use case is taking a server out of the cluster for maintenance. This +is done by gracefully failing over a server to ensure there is no data loss. +The administrator can then perform maintenance on the removed server and add it +back with the server-readd command.

    +

    The recovery command also allows a server to have it’s data remvoed before being +added back to the cluster (full recovery) or having the server resume from where +it last left off (delta recovery). Delta recovery will always take the least +amount of time and is the recommended recovery mode.

    +

    Note that after the recovery subcommand is run you must rebalance the +cluster. See the couchbase-rebalance(1) command for more information +on rebalancing a cluster.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--server-recovery <servers> +
    +
    +

    + A command separated list of servers to recover. The each server should + correspond to the hostname or IP address of a server to be added back to + the cluster. +

    +
    +
    +--recovery-type <type> +
    +
    +

    + Specifies whether or not we should do full or delta recovery of data on the + servers being added back. Full recover means that all data on the servers + being recovered will be removed and we will recover all data. Specifying + delta recover means we will keep the data that is on each server and only + recover the data that each server doesn’t have. To specify full recovery set + this option to "full". For delta recovery set this option to "delta". The + default value for this option is "full". +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    If we have a server at 192.168.1.6:8091 that we want to recover and we want to +recover data using delta recovery then we can run the following command.

    +
    +
    +
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-recovery 192.168.1.6:8091 \
+ --recovery-type delta
    +
    +

    If we need to recover multiple servers then we would run the following command.

    +
    +
    +
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-recovery 192.168.1.6:8091,192.168.1.7:8091 \
+ --recovery-type delta
    +
    +

    If we wanted the servers being recovered to use full recovery then we would run +the following command.

    +
    +
    +
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-recovery 192.168.1.6:8091 \
+ --recovery-type full
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-rebalance(1). +couchbase-server-add(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-reset-admin-password.dita b/content/cli/cbcli/couchbase-cli-reset-admin-password.dita deleted file mode 100644 index ec2c0923e7..0000000000 --- a/content/cli/cbcli/couchbase-cli-reset-admin-password.dita +++ /dev/null @@ -1,119 +0,0 @@ - - - reset-admin-password - - Resets the Couchbase Server administrator password
    SYNTAX - - - couchbase-cli reset-admin-password [--cluster <url>] [--regenerate] - [--new-password <password>]
    DESCRIPTION - - -

    This command is used to reset the built-in Couchbase Server Administrator password. -There is only one built-in Administrator account, which is managed separately -from the internal and external Administrator users. -External and Internal Administrator -accounts can be managed using the -couchbase-cli-user-manage1 - command.

    -

    The reset-admin-password command must be run locally on a node that is part of the -cluster (i.e. must connect to the host the command is run on). -This is required as reset-admin-password -relies on a local authentication token, which is used to authenticate with the -server and change the password. As a result, the command does not require -credentials to be passed.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    ---new-password <password> -
    -
    -

    - Sets the password for the Couchbase Server administrator user to the value - specified by the argument. If no password is - specified, the command prompts the user for the new - password through non-echoed stdin. -

    -
    -     - -
    ---regenerate -
    -
    -

    - Sets the password for the Couchbase Server administrator user to a randomly - generated value. The new password is printed to the command line after - the password is changed. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To change the administrator password to new_pwd run the following command:

    - 
    $ couchbase-cli reset-admin-password -c 192.168.1.5:8091 \
- --new-password new_pwd
    -

    To change the administrator password to a randomly generated value, run the -following command. The new password will be printed to stdout if the -password is successfully changed:

    - 
    $ couchbase-cli reset-admin-password -c 192.168.1.5:8091 --regenerate
-  jXjNW6LG
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-reset-admin-password.html b/content/cli/cbcli/couchbase-cli-reset-admin-password.html new file mode 100644 index 0000000000..1f12ceb840 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-reset-admin-password.html @@ -0,0 +1,847 @@ + + + + + +couchbase-cli-reset-admin-password(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli reset-admin-password [--regenerate] [--new-password <password>]
+          [--port <port>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command is used to reset the built-in Couchbase Server Administrator +password. There is only one built-in Administrator account, which is managed +separately from the internal and external Administrator users. External and +internal Administrator accounts can be managed using the +couchbase-cli-user-manage(1) command.

    +

    The reset-admin-password command must be run locally on a node that is part +of the cluster (i.e. must connect to the host the command is run on). This is +required, as reset-admin-password relies on a local authentication token, +which is used to authenticate with the server and change the password. As a +result, the command does not require credentials to be passed.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +--new-password <password> +
    +
    +

    + Sets the password for the Couchbase Server administrator user to the value + specified by the argument. If no password is specified, the command + prompts the user for the new password through non-echoed stdin. +

    +
    +
    +--regenerate +
    +
    +

    + Sets the password for the Couchbase Server administrator user to a randomly + generated value. The new password is printed to the command line after + the password is changed. +

    +
    +
    +--port +
    +
    +

    + Specify the REST API port of the locally running Couchbase Server. If no + port is specified the default port 8091 is used. +

    +
    +
    +
    +
    +
    +

    EXAMPLES

    +
    +

    To change the administrator password to new_pwd, run the following command:

    +
    +
    +
    $ couchbase-cli reset-admin-password --new-password new_pwd
    +
    +

    To change the administrator password to a randomly generated value, run the +following command. The new password will be printed to stdout if the +password is successfully changed:

    +
    +
    +
    $ couchbase-cli reset-admin-password --regenerate
+  jXjNW6LG
    +
    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-ldap(1). +couchbase-cli-user-manage(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-server-add.dita b/content/cli/cbcli/couchbase-cli-server-add.dita deleted file mode 100644 index 33b6a7e85e..0000000000 --- a/content/cli/cbcli/couchbase-cli-server-add.dita +++ /dev/null @@ -1,206 +0,0 @@ - - - server-add - - Adds a server to the cluster
    SYNTAX - - - couchbase-cli server-add [--cluster <url>] [--username <username>] - [--password <password>] [--server-add <servers>] [--group-name <name>] - [--server-add-username <username>] [--server-add-password <password>] - [--services <services>] [--index-storage-setting <mode>]
    DESCRIPTION - - -

    The server-add subcommand is used to add one or more servers to a cluster. -Before adding a server it is important to decide which services the server will -be running and whether or not the server should be a part of a specific group. -Keep in mind that if the index service is being added on one of the servers and -the cluster is not currently running the index service that you also need to -set the index stoare mode. This can be done with the --index-storage-setting -option.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---server-add <servers> -
    -
    -

    - A comma separated list of servers to add to the cluster. The each server in - the list should be indentified by a hostname or IP address. -

    -
    -     - -
    ---server-add-username <username> -
    -
    -

    - Adding a server to the cluster can only be done by a user with the - appropriate credentials. This flag specifies the username for a user who - has the ability to modify the cluster topology on the server being added. -

    -
    -     - -
    ---server-add-password <password> -
    -
    -

    - Adding a server to the cluster can only be done by a user with the - appropriate credentials. This flag specifies the password for a user who - has the ability to modify the cluster topology on the server being added. -

    -
    -     - -
    ---group-name <name> -
    -
    -

    - The name of the group to add the server to. If this option is not specified - then the server is added to the default group. -

    -
    -     - -
    ---services <services> -
    -
    -

    - A comma separated list of services that this server should be running. The - data service is specified with "data", the index service is specified with - "index", the query service is specified with "query", and the full-text - service is specified with "fts". -

    -
    -     - -
    ---index-storage-setting <mode> -
    -
    -

    - Specifies the index storage mode. This parameter must be set if the servers - being added contain the index service and this is the first time the index - service is being added in this cluster. You may specify "default" for disk - based indexes or "memopt" for memory optimized indexes. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    If we want to add a server at 192.168.1.6:8091 with the index, data and query -service on it then we would run the command below.

    - 
    $ couchbase-cli server-add -c 192.168.1.5:8091 --username Administrator \
- --password password --server-add 192.168.1.6:8091 \
- --server-add-username Administrator --server-add-password password
- --services data,index,query
    -

    Note that in the example above we set the username and password of the server -being added to the same value as the username and password of the servers -already in the cluster. This is done if the server being added has not already -been initialized. If the server being added has been initialized then you will -need to specify an appropriate username and password for the server being added.

    -

    Now lets add two server to the East group with data, index,and full-text -services. In this example we will also assume that the index is being added for -the first time so we need to specify the index storage mode. If we want to index -storage mode to be memory optimized then we would run the following command.

    - 
    $ couchbase-cli server-add -c 192.168.1.5:8091 --username Administrator \
- --password password --server-readd 192.168.1.6:8091,192.168.1.7:8091 \
- --server-username Administrator --server-password password
- --services data,fts --group-name --index-storage-setting memopt
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-server-add.html b/content/cli/cbcli/couchbase-cli-server-add.html new file mode 100644 index 0000000000..1f9090c7b0 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-server-add.html @@ -0,0 +1,972 @@ + + + + + +couchbase-cli-server-add(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli server-add [--cluster <url>] [--username <username>]
+          [--password <password>] [--server-add <servers>] [--group-name <name>]
+          [--server-add-username <username>] [--server-add-password <password>]
+          [--services <services>] [--index-storage-setting <mode>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    The server-add subcommand is used to add one or more servers to a cluster. +Before adding a server it is important to decide which services the server will +be running and whether or not the server should be a part of a specific group. +Keep in mind that if the index service is being added on one of the servers and +the cluster is not currently running the index service that you also need to +set the index stoare mode. This can be done with the --index-storage-setting +option.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--server-add <servers> +
    +
    +

    + A comma separated list of servers to add to the cluster. The each server in + the list should be identified by a hostname or IP address. +

    +
    +
    +--server-add-username <username> +
    +
    +

    + Adding a server to the cluster can only be done by a user with the + appropriate credentials. This flag specifies the username for a user who + has the ability to modify the cluster topology on the server being added. +

    +
    +
    +--server-add-password <password> +
    +
    +

    + Adding a server to the cluster can only be done by a user with the + appropriate credentials. This flag specifies the password for a user who + has the ability to modify the cluster topology on the server being added. +

    +
    +
    +--group-name <name> +
    +
    +

    + The name of the group to add the server to. If this option is not specified + then the server is added to the default group. +

    +
    +
    +--services <services> +
    +
    +

    + A comma separated list of services that this server should be running. The + data service is specified with "data", the index service is specified with + "index", the query service is specified with "query", and the full-text + service is specified with "fts". +

    +
    +
    +--index-storage-setting <mode> +
    +
    +

    + Specifies the index storage mode. This parameter must be set if the servers + being added contain the index service and this is the first time the index + service is being added in this cluster. You may specify "default" for disk + based indexes or "memopt" for memory optimized indexes. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    If we want to add a server at 192.168.1.6:8091 with the index, data and query +service on it then we would run the command below.

    +
    +
    +
    $ couchbase-cli server-add -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-add 192.168.1.6:8091 \
+ --server-add-username Administrator --server-add-password password
+ --services data,index,query
    +
    +

    Note that in the example above we set the username and password of the server +being added to the same value as the username and password of the servers +already in the cluster. This is done if the server being added has not already +been initialized. If the server being added has been initialized then you will +need to specify an appropriate username and password for the server being added.

    +

    Now lets add two server to the East group with data, index,and full-text +services. In this example we will also assume that the index is being added for +the first time so we need to specify the index storage mode. If we want to index +storage mode to be memory optimized then we would run the following command.

    +
    +
    +
    $ couchbase-cli server-add -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-readd 192.168.1.6:8091,192.168.1.7:8091 \
+ --server-username Administrator --server-password password
+ --services data,fts --group-name --index-storage-setting memopt
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-rebalance(1). +couchbase-server-info(1). +couchbase-server-list(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-server-eshell.html b/content/cli/cbcli/couchbase-cli-server-eshell.html new file mode 100644 index 0000000000..332c55e793 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-server-eshell.html @@ -0,0 +1,937 @@ + + + + + +couchbase-cli-server-eshell(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli server-eshell [--cluster <url>] [--username <user>]
+          [--password <password>] [--vm <vm_name>] [--erl <path>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This is a hidden command and is not intended for typical production use. This +command allows the user to connect to the cluster manager process in order to +make unsupported changes in the cluster manager. This command is typically used +as last resort in order to address uncommon production issues. Check with the +Couchbase Support team before running this command because improper use can lead +to the cluster being placed in an unusable state.

    +

    When this command is executed it connects to the cluster manager process +specified and starts a shell so that the user can interact with the cluster +manager.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--vm +
    +
    +

    + The part of the cluster manager to connect to. This option can either be set + to babysitter, couchdb, or ns_server. The babysitter vm is a monitor that + ensures the cluster manager is always running and is restarted in the event + of a crash or error. The couchdb vm runs the views service on each data node + and the ns_server vm corresponds to the cluster manager. By default the vm + option is set to ns_server. +

    +
    +
    +--erl +
    +
    +

    + Specified the location of the erlang process. This parameter should be set + if the default erlang process that is shipped with Couchbase is not the + correct executable to connect to the cluster manager with. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To connect to the cluster manager process you can run the following command +below.

    +
    +
    +
    $ couchbase-cli server-eshell -c 192.168.1.5:8091 --username Administrator \
+ --password password
    +
    +
    +
    +
    Erlang R16B03-1 (erts-5.10.4.0.0.1)
    +
    +
    +
    +
    Eshell V5.10.4.0.0.1  (abort with ^G)
+(n_0@192.168.1.5)1>
    +
    +

    When the command is executed it will bring up a shell that is connected to the +vm specified in the command. Since the --vm parameter is not specified we +connect to the default ns_server vm.

    +

    To connect to the babysitter vm the following command would be run.

    +
    +
    +
    $ couchbase-cli server-eshell -c 192.168.1.5:8091 --username Administrator \
+ --password password --vm babysitter
    +
    +
    +
    +
    Erlang R16B03-1 (erts-5.10.4.0.0.1)
    +
    +
    +
    +
    Eshell V5.10.4.0.0.1  (abort with ^G)
+(n_0@192.168.1.5)1>
    +
    +

    To exit the shell send the SIGINT signal. This can be done by typing Ctrl-C.

    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-server-info.dita b/content/cli/cbcli/couchbase-cli-server-info.dita deleted file mode 100644 index 1cc91d0ae4..0000000000 --- a/content/cli/cbcli/couchbase-cli-server-info.dita +++ /dev/null @@ -1,112 +0,0 @@ - - - server-info - - Displays information and statistics about -the Couchbase Server specified.
    SYNTAX - - - couchbase-cli server-info [--cluster <url>] [--username <username>] - [--password <password>]
    DESCRIPTION - - -

    This command prints information about the Couchbase Server specified.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To get information about the Couchbase Server on localhost (127.0.0.1) run -the following command:

    - 
    $ couchbase-cli server-info -c 127.0.0.1 --username Administrator \
- --password password
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-server-info.html b/content/cli/cbcli/couchbase-cli-server-info.html new file mode 100644 index 0000000000..48e83b81a2 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-server-info.html @@ -0,0 +1,885 @@ + + + + + +couchbase-cli-server-info(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli server-info [--cluster <url>] [--username <username>]
+          [--password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command prints information about the Couchbase Server specified.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To get information about the Couchbase Server on localhost (127.0.0.1) run +the following command:

    +
    +
    +
    $ couchbase-cli server-info -c 127.0.0.1 --username Administrator \
+ --password password
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-host-list(1). +couchbase-cli-server-list(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-server-list.dita b/content/cli/cbcli/couchbase-cli-server-list.dita deleted file mode 100644 index 612ce6559c..0000000000 --- a/content/cli/cbcli/couchbase-cli-server-list.dita +++ /dev/null @@ -1,115 +0,0 @@ - - - server-list - - Lists all servers in the cluster
    SYNTAX - - - couchbase-cli server-list [--cluster <url>] [--username <user>] - [--password <password>]
    DESCRIPTION - - -

    Returns a new line separated list of all servers in the cluster. Each server -name will be either a hostname or IP address. If no port number is specified for -the server then the port number is 8091.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To get a list of servers in the cluster you can run the command below.

    - 
    $ couchbase-cli server-list -c 192.168.1.5:8091 --username Administrator \
- --password password
    - 
    192.168.1.5:8091
-192.168.1.6:8091
-192.168.1.7:8091
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-server-list.html b/content/cli/cbcli/couchbase-cli-server-list.html new file mode 100644 index 0000000000..0e1b885c74 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-server-list.html @@ -0,0 +1,892 @@ + + + + + +couchbase-cli-server-list(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli server-list [--cluster <url>] [--username <user>]
+          [--password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Returns a new line separated list of all servers in the cluster. Each server +name will be either a hostname or IP address. If no port number is specified for +the server then the port number is 8091.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To get a list of servers in the cluster you can run the command below.

    +
    +
    +
    $ couchbase-cli server-list -c 192.168.1.5:8091 --username Administrator \
+ --password password
    +
    +
    +
    +
    192.168.1.5:8091
+192.168.1.6:8091
+192.168.1.7:8091
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-host-list(1) +couchbase-cli-server-info(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-server-readd.dita b/content/cli/cbcli/couchbase-cli-server-readd.dita deleted file mode 100644 index ea256edfe1..0000000000 --- a/content/cli/cbcli/couchbase-cli-server-readd.dita +++ /dev/null @@ -1,198 +0,0 @@ - - - server-readd - - Adds a node back to the cluster after a failover
    SYNTAX - - - couchbase-cli server-readd [--cluster <url>] [--username <username>] - [--password <password>] [--server-add <servers>] [--group-name <name>] - [--server-username <username>] [--server-password <password>]
    DESCRIPTION - - -

    DEPRECATED: This command was deprecated in 5.0.0 and will be removed in future -releases. Please use the -couchbase-cli-recovery1 - subcommand which -provides similar functionality to this command.

    -

    The server-readd subcommand is used to add a server back to the cluster. This -operation is useful after a node is failed over and also when a node is -removed from the cluster for maintainance.

    -

    When a node is failed over and removed from the cluster it may be able to -be added back to the cluster. An example of this is when a node loses power. -This node might get failed over and removed from the cluster, but once power -is restored to the node you may want to add it back to the cluster.

    -

    Another use case is taking a node out of the cluster for maintainance. This -is done by gracefully failing over a node to ensure there is no data loss. -The administrator can then perform maintenance on the removed node and add it -back with the server-readd command.

    -

    It is also possible to add a server back to the cluster without removing the -data and instead having the server recover data from where it left off. This is -called delta recovery and is available from the - -couchbase-cli-recovery1 - subcommand.

    -

    Note that after the server-readd subcommand is run you must rebalance the -cluster. See the -couchbase-cli-rebalance1 - command for more -information on rebalancing a cluster.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---server-add <servers> -
    -
    -

    - A comma separated list of nodes to readd. The each server should - correspond to the hostname or IP address of a server to be added back to - the cluster. -

    -
    -     - -
    ---server-username <username> -
    -
    -

    - Adding a node back to the cluster can only be done by a user with the - appropriate credentials. This flag specifies the username for a user who - has the ability to modify the cluster topology of the node being added - back. -

    -
    -     - -
    ---server-password <password> -
    -
    -

    - Adding a node back to the cluster can only be done by a user with the - appropriate credentials. This flag specifies the password for a user who - has the ability to modify the cluster topology of the node being added - back. -

    -
    -     - -
    ---group-name <name> -
    -
    -

    - The name of the group to add the node to. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    If there is a node at 192.168.1.6:8091 that you want to add back and that -node has credentials that allow us to change the cluster topology with -"Administrator" as the username and "password" as the password, run the -following command.

    - 
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
- --password password --server-readd 192.168.1.6:8091 \
- --server-username Administrator --server-password password
    -

    If you need to add back multiple nodes then you would run the following command.

    - 
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
- --password password --server-readd 192.168.1.6:8091,192.168.1.7:8091 \
- --server-username Administrator --server-password password
    -

    If you wanted the nodes being added back to the cluster to be a part of the -"East" group we would run the following command

    - 
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
- --password password --server-readd 192.168.1.6:8091 --group-name East \
- --server-username Administrator --server-password password
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-server-readd.html b/content/cli/cbcli/couchbase-cli-server-readd.html new file mode 100644 index 0000000000..c3e444abc3 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-server-readd.html @@ -0,0 +1,965 @@ + + + + + +couchbase-cli-server-readd(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli server-readd [--cluster <url>] [--username <username>]
+          [--password <password>] [--server-add <servers>] [--group-name <name>]
+          [--server-username <username>] [--server-password <password>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    DEPRECATED: This command was deprecated in 5.0.0 and will be removed in future +releases. Please use the couchbase-cli-recovery(1) subcommand which +provides similar functionality to this command.

    +

    The server-readd subcommand is used to add a server back to the cluster. This +operation is useful after a node is failed over and also when a node is +removed from the cluster for maintenance.

    +

    When a node is failed over and removed from the cluster it may be able to +be added back to the cluster. An example of this is when a node loses power. +This node might get failed over and removed from the cluster, but once power +is restored to the node you may want to add it back to the cluster.

    +

    Another use case is taking a node out of the cluster for maintenance. This +is done by gracefully failing over a node to ensure there is no data loss. +The administrator can then perform maintenance on the removed node and add it +back with the server-readd command.

    +

    It is also possible to add a server back to the cluster without removing the +data and instead having the server recover data from where it left off. This is +called delta recovery and is available from the +couchbase-cli-recovery(1) subcommand.

    +

    Note that after the server-readd subcommand is run you must rebalance the +cluster. See the couchbase-cli-rebalance(1) command for more +information on rebalancing a cluster.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--server-add <servers> +
    +
    +

    + A comma separated list of nodes to readd. The each server should + correspond to the hostname or IP address of a server to be added back to + the cluster. +

    +
    +
    +--server-username <username> +
    +
    +

    + Adding a node back to the cluster can only be done by a user with the + appropriate credentials. This flag specifies the username for a user who + has the ability to modify the cluster topology of the node being added + back. +

    +
    +
    +--server-password <password> +
    +
    +

    + Adding a node back to the cluster can only be done by a user with the + appropriate credentials. This flag specifies the password for a user who + has the ability to modify the cluster topology of the node being added + back. +

    +
    +
    +--group-name <name> +
    +
    +

    + The name of the group to add the node to. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    If there is a node at 192.168.1.6:8091 that you want to add back and that +node has credentials that allow us to change the cluster topology with +"Administrator" as the username and "password" as the password, run the +following command.

    +
    +
    +
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-readd 192.168.1.6:8091 \
+ --server-username Administrator --server-password password
    +
    +

    If you need to add back multiple nodes then you would run the following command.

    +
    +
    +
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-readd 192.168.1.6:8091,192.168.1.7:8091 \
+ --server-username Administrator --server-password password
    +
    +

    If you wanted the nodes being added back to the cluster to be a part of the +"East" group we would run the following command

    +
    +
    +
    $ couchbase-cli server-readd -c 192.168.1.5:8091 --username Administrator \
+ --password password --server-readd 192.168.1.6:8091 --group-name East \
+ --server-username Administrator --server-password password
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-rebalance(1). +couchbase-cli-recovery(1). +couchbase-cli-server-add(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-alert.dita b/content/cli/cbcli/couchbase-cli-setting-alert.dita deleted file mode 100644 index 0307e28661..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-alert.dita +++ /dev/null @@ -1,371 +0,0 @@ - - - setting-alert - - Modifies alert settings
    SYNTAX - - - couchbase-cli setting-alert [--cluster <url>] [--username <user>] - [--password <password>] [--enable-email-alert <num>] - [--email-recipients <email_list>] [--email-sender <email>] - [--email-user <user>] [--email-password <password>] [--email-host <host>] - [--email-port <port>] [--enable-email-encrypt <num>] - [--alert-auto-failover-node] [--alert-auto-failover-max-reached] - [--alert-auto-failover-node-down] [--alert-auto-failover-cluster-small] - [--alert-auto-failover-disable] [--alert-ip-changed] [--alert-disk-space] - [--alert-meta-overhead] [--alert-meta-oom] [--alert-write-failed] - [--alert-audit-msg-dropped] [--alert-indexer-max-ram] - [--alert-timestamp-drift-exceeded]
    DESCRIPTION - - -

    This command is used to set up email alerts on a cluster. Couchbase provides -alerts for various issues that may arise in the cluster where it is recommended -that the cluster administrator take action to ensure that applications continue -to function properly. When setting up email alerts administrator can decide who -gets alert emails and which alerts are sent.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---enable-email-alert <num> -
    -
    -

    - Enables email alerts on this cluster. Set this option to "1" to enable - alerts or "0" to disable alerts. This parameter is required. -

    -
    -     - -
    ---email-recipients <email_list> -
    -
    -

    - A comma separated list of users to email when an alert is raised in the - server. -

    -
    -     - -
    ---email-sender <email> -
    -
    -

    - If email alerts an enabled then this option will set the sender email - address. -

    -
    -     - -
    ---email-user <user> -
    -
    -

    - The email server username for the sender email address. This field is - required if the email address provided requires authentication. -

    -
    -     - -
    ---email-password <password> -
    -
    -

    - The email server password for the sender email address. This field is - required if the email address provided requires authentication. -

    -
    -     - -
    ---email-host <host> -
    -
    -

    - The email server hostname that hosts the email address specified by the - --sender-email option. -

    -
    -     - -
    ---email-port <port> -
    -
    -

    - The email server port number of the server that hosts the email address - specified by the --sender-email option. -

    -
    -     - -
    ---enable-email-encrypt <num> -
    -
    -

    - Enables SSL encryption when connecting to the email server. Set this option - to "1" to enable encryption or "0" to disable encryption. If this option is - not set then encryption will be disabled. -

    -
    -     - -
    ---alert-auto-failover-node -
    -
    -

    - Specifies that an email alert should be sent when a node is automatically - failed over. -

    -
    -     - -
    ---alert-auto-failover-max-reached -
    -
    -

    - Specifies that an email alert should be sent when the maximum amount of - auto-failovers is reached. -

    -
    -     - -
    ---alert-auto-failover-node-down -
    -
    -

    - Specifies that an email alert should be sent when auto-failover could not be - completed because another node in the cluster was already down. -

    -
    -     - -
    ---alert-auto-failover-cluster-small -
    -
    -

    - Specifies that an email alert should be sent when auto-failover could not be - completed because the cluster is too small. -

    -
    -     - -
    ---alert-auto-failover-disable -
    -
    -

    - Specifies that an email alert should be sent when auto-failover could not be - completed because auto-failover is disabled on this cluster. -

    -
    -     - -
    ---alert-ip-changed -
    -
    -

    - Specifies that an email alert should be sent when the IP address on a node - in the cluster changes. -

    -
    -     - -
    ---alert-disk-space -
    -
    -

    - Specifies that an email alert should be sent when the disk usage on a node - in the cluster reaches 90% of the available disk space. -

    -
    -     - -
    ---alert-meta-overhead -
    -
    -

    - Specifies that an email alert should be sent when the metadata overhead on - the data service is more than 50%. -

    -
    -     - -
    ---alert-meta-oom -
    -
    -

    - Specifies that an email alert should be sent when all of the memory in the - cache for a bucket is used by metadata. If this condition is hit the bucket - will be unusable until more memory is added to the bucket cache. -

    -
    -     - -
    ---alert-write-failed -
    -
    -

    - Specifies that an email alert should be sent when writing data to disk on - the data service has failed. -

    -
    -     - -
    ---alert-audit-msg-dropped -
    -
    -

    - Specifies that an email alert should be sent when writing event to audit log - fails. -

    -
    -     - -
    ---alert-indexer-max-ram -
    -
    -

    - Specifies that an email alert should be sent when the memory usage for the - indexer service on a specific node exceeds the per node memory usage limit. - This warning is only shown for if the index storage type is Memory Optimized - Indexes (MOI). -

    -
    -     -
    -

    --alert-timestamp-drift-exceeded - Specifies that an email alert should be sent if the clocks on two different - machines in the cluster are more that five seconds apart.

    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To enable failover related email alerts for two users without email encryption -run the following command below:

    - 
    $ couchbase-cli setting-alert -c 192.168.1.5 --username Administrator \
- --password password --enable-email-alert 1 --email-user admin \
- --email-password password --email-host mail.couchbase.com --email-port 25 \
- --email-recipients user1@couchbase.com,user2@couchbase.com \
- --email-sender noreply@couchbase.com --enable-email-encrypt 0 \
- --alert-auto-failover-node --alert-auto-failover-max-reached \
- --alert-auto-failover-node-down --alert-auto-failover-cluster-small \
    -

    To enable all email alerts for a single users with email encryption run the -following command below:

    - 
    $ couchbase-cli setting-alert -c 192.168.1.5 --username Administrator \
- --password password --enable-email-alert 1 --email-user admin \
- --email-password password --email-host mail.couchbase.com --email-port 25 \
- --email-recipients user@couchbase.com --email-sender noreply@couchbase.com \
- --enable-email-encrypt 1 --alert-auto-failover-node \
- --alert-auto-failover-max-reached --alert-auto-failover-node-down \
- --alert-auto-failover-cluster-small --alert-auto-failover-disable \
- --alert-ip-changed --alert-disk-space --alert-meta-overhead \
- --alert-meta-oom --alert-write-failed --alert-audit-msg-dropped \
- --alert-indexer-max-ram --alert-timestamp-drift-exceeded \
    -

    To disable email alerts run the following command:

    - 
    $ couchbase-cli setting-alert -c 192.168.1.5 --username Administrator \
- --password password --enable-email-alert 0
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-alert.html b/content/cli/cbcli/couchbase-cli-setting-alert.html new file mode 100644 index 0000000000..1b69390ecd --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-alert.html @@ -0,0 +1,1110 @@ + + + + + +couchbase-cli-setting-alert(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-alert [--cluster <url>] [--username <user>]
+          [--password <password>] [--enable-email-alert <num>]
+          [--email-recipients <email_list>] [--email-sender <email>]
+          [--email-user <user>] [--email-password <password>] [--email-host <host>]
+          [--email-port <port>] [--enable-email-encrypt <num>]
+          [--alert-auto-failover-node] [--alert-auto-failover-max-reached]
+          [--alert-auto-failover-node-down] [--alert-auto-failover-cluster-small]
+          [--alert-auto-failover-disable] [--alert-ip-changed] [--alert-disk-space]
+          [--alert-meta-overhead] [--alert-meta-oom] [--alert-write-failed]
+          [--alert-audit-msg-dropped] [--alert-indexer-max-ram]
+          [--alert-timestamp-drift-exceeded]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command is used to set up email alerts on a cluster. Couchbase provides +alerts for various issues that may arise in the cluster where it is recommended +that the cluster administrator take action to ensure that applications continue +to function properly. When setting up email alerts administrator can decide who +gets alert emails and which alerts are sent.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--enable-email-alert <num> +
    +
    +

    + Enables email alerts on this cluster. Set this option to "1" to enable + alerts or "0" to disable alerts. This parameter is required. +

    +
    +
    +--email-recipients <email_list> +
    +
    +

    + A comma separated list of users to email when an alert is raised in the + server. +

    +
    +
    +--email-sender <email> +
    +
    +

    + If email alerts an enabled then this option will set the sender email + address. +

    +
    +
    +--email-user <user> +
    +
    +

    + The email server username for the sender email address. This field is + required if the email address provided requires authentication. +

    +
    +
    +--email-password <password> +
    +
    +

    + The email server password for the sender email address. This field is + required if the email address provided requires authentication. +

    +
    +
    +--email-host <host> +
    +
    +

    + The email server hostname that hosts the email address specified by the + --sender-email option. +

    +
    +
    +--email-port <port> +
    +
    +

    + The email server port number of the server that hosts the email address + specified by the --sender-email option. +

    +
    +
    +--enable-email-encrypt <num> +
    +
    +

    + Enables SSL encryption when connecting to the email server. Set this option + to "1" to enable encryption or "0" to disable encryption. If this option is + not set then encryption will be disabled. +

    +
    +
    +--alert-auto-failover-node +
    +
    +

    + Specifies that an email alert should be sent when a node is automatically + failed over. +

    +
    +
    +--alert-auto-failover-max-reached +
    +
    +

    + Specifies that an email alert should be sent when the maximum amount of + auto-failovers is reached. +

    +
    +
    +--alert-auto-failover-node-down +
    +
    +

    + Specifies that an email alert should be sent when auto-failover could not be + completed because another node in the cluster was already down. +

    +
    +
    +--alert-auto-failover-cluster-small +
    +
    +

    + Specifies that an email alert should be sent when auto-failover could not be + completed because the cluster is too small. +

    +
    +
    +--alert-auto-failover-disable +
    +
    +

    + Specifies that an email alert should be sent when auto-failover could not be + completed because auto-failover is disabled on this cluster. +

    +
    +
    +--alert-ip-changed +
    +
    +

    + Specifies that an email alert should be sent when the IP address on a node + in the cluster changes. +

    +
    +
    +--alert-disk-space +
    +
    +

    + Specifies that an email alert should be sent when the disk usage on a node + in the cluster reaches 90% of the available disk space. +

    +
    +
    +--alert-meta-overhead +
    +
    +

    + Specifies that an email alert should be sent when the metadata overhead on + the data service is more than 50%. +

    +
    +
    +--alert-meta-oom +
    +
    +

    + Specifies that an email alert should be sent when all of the memory in the + cache for a bucket is used by metadata. If this condition is hit the bucket + will be unusable until more memory is added to the bucket cache. +

    +
    +
    +--alert-write-failed +
    +
    +

    + Specifies that an email alert should be sent when writing data to disk on + the data service has failed. +

    +
    +
    +--alert-audit-msg-dropped +
    +
    +

    + Specifies that an email alert should be sent when writing event to audit log + fails. +

    +
    +
    +--alert-indexer-max-ram +
    +
    +

    + Specifies that an email alert should be sent when the memory usage for the + indexer service on a specific node exceeds the per node memory usage limit. + This warning is only shown for if the index storage type is Memory Optimized + Indexes (MOI). +

    +
    +
    +

    --alert-timestamp-drift-exceeded + Specifies that an email alert should be sent if the clocks on two different + machines in the cluster are more that five seconds apart.

    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To enable failover related email alerts for two users without email encryption +run the following command below:

    +
    +
    +
    $ couchbase-cli setting-alert -c 192.168.1.5 --username Administrator \
+ --password password --enable-email-alert 1 --email-user admin \
+ --email-password password --email-host mail.couchbase.com --email-port 25 \
+ --email-recipients user1@couchbase.com,user2@couchbase.com \
+ --email-sender noreply@couchbase.com --enable-email-encrypt 0 \
+ --alert-auto-failover-node --alert-auto-failover-max-reached \
+ --alert-auto-failover-node-down --alert-auto-failover-cluster-small \
    +
    +

    To enable all email alerts for a single users with email encryption run the +following command below:

    +
    +
    +
    $ couchbase-cli setting-alert -c 192.168.1.5 --username Administrator \
+ --password password --enable-email-alert 1 --email-user admin \
+ --email-password password --email-host mail.couchbase.com --email-port 25 \
+ --email-recipients user@couchbase.com --email-sender noreply@couchbase.com \
+ --enable-email-encrypt 1 --alert-auto-failover-node \
+ --alert-auto-failover-max-reached --alert-auto-failover-node-down \
+ --alert-auto-failover-cluster-small --alert-auto-failover-disable \
+ --alert-ip-changed --alert-disk-space --alert-meta-overhead \
+ --alert-meta-oom --alert-write-failed --alert-audit-msg-dropped \
+ --alert-indexer-max-ram --alert-timestamp-drift-exceeded \
    +
    +

    To disable email alerts run the following command:

    +
    +
    +
    $ couchbase-cli setting-alert -c 192.168.1.5 --username Administrator \
+ --password password --enable-email-alert 0
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-notification(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-audit.dita b/content/cli/cbcli/couchbase-cli-setting-audit.dita deleted file mode 100644 index a7a20658a9..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-audit.dita +++ /dev/null @@ -1,157 +0,0 @@ - - - setting-audit - - Modifies audit log settings
    SYNTAX - - - couchbase-cli setting-audit [--cluster <url>] [--username <user>] - [--password <password>] [--audit-enabled <num>] [--audit-log-path <path>] - [--audit-log-rotate-interval <seconds>]
    DESCRIPTION - - -

    Auditing is used to observe the action of users in the cluster. It is usually -turned on for security purposes to ensure that those without permission to -access information do not access it. By default auditing is disabled in -Couchbase. The setting-audit command can be used to enable auditing, set the -auditing log path, and change the auditing log rotation interval.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---audit-enabled <num> -
    -
    -

    - Specifies whether or not auditing is enabled. To enabled auditing set this - option to "1". To disable auditing set this option to "0". -

    -
    -     - -
    ---audit-log-path <path> -
    -
    -

    - Specifies The auditing log path. This should be a path should be a path to a - folder where the auditing log is kept. The folder must exist on all servers - in the cluster. -

    -
    -     - -
    ---audit-log-rotate-interval <seconds> -
    -
    -

    - Specifies the audit log rotate interval. This is the interval in which the - current audit log will be replaced with a new empty audit log file. The log - file is rotated to ensure that the audit log does not consume too much disk - space. The minimum audit log rotate interval is 15 minutes (900 seconds). -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To enable auditing on a cluster and set the audit log to the default logs folder -on a linux installation with a log rotate interval of 7 days run the command -below.

    - 
    $ couchbase-cli setting-audit -c 192.168.1.5:8091 --username Administrator \
- --password password --enabled 1 --audit-log-rotate-interval 604800 \
- --audit-log-path /opt/couchbase/var/lib/couchbase/logs
    -

    To disable auditing run the following command.

    - 
    $ couchbase-cli setting-audit -c 192.168.1.5:8091 --username Administrator \
- --password password --enabled 1
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-audit.html b/content/cli/cbcli/couchbase-cli-setting-audit.html new file mode 100644 index 0000000000..f6c8243f35 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-audit.html @@ -0,0 +1,949 @@ + + + + + +couchbase-cli-setting-audit(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-audit [--cluster <url>] [--username <user>]
+          [--password <password>] [--audit-enabled <num>] [--audit-log-path <path>]
+          [--audit-log-rotate-interval <seconds>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Auditing is used to observe the action of users in the cluster. It is usually +turned on for security purposes to ensure that those without permission to +access information do not access it. By default auditing is disabled in +Couchbase. The setting-audit command can be used to enable auditing, set the +auditing log path, and change the auditing log rotation interval.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--audit-enabled <num> +
    +
    +

    + Specifies whether or not auditing is enabled. To enabled auditing set this + option to "1". To disable auditing set this option to "0". +

    +
    +
    +--audit-log-path <path> +
    +
    +

    + Specifies The auditing log path. This should be a path should be a path to a + folder where the auditing log is kept. The folder must exist on all servers + in the cluster. +

    +
    +
    +--audit-log-rotate-interval <seconds> +
    +
    +

    + Specifies the audit log rotate interval. This is the interval in which the + current audit log will be replaced with a new empty audit log file. The log + file is rotated to ensure that the audit log does not consume too much disk + space. The minimum audit log rotate interval is 15 minutes (900 seconds). +

    +
    +
    +--audit-log-rotate-size <bytes> +
    +
    +

    + Specifies the audit log rotate size. This is the size at which the current + audit log will be replaced with a new empty audit log file. The log + file is rotated to ensure that the audit log does not consume too much disk + space. The minimum audit log rotate size is 0 bytes and maximum is + 524,288,000 (500MB). When it is set to 0 the log will not be rotated based + on size. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To enable auditing on a cluster and set the audit log to the default logs folder +on a linux installation with a log rotate interval of 7 days run the command +below.

    +
    +
    +
    $ couchbase-cli setting-audit -c 192.168.1.5:8091 --username Administrator \
+ --password password --enabled 1 --audit-log-rotate-interval 604800 \
+ --audit-log-path /opt/couchbase/var/lib/couchbase/logs
    +
    +

    To enable auditing on a cluster and set the audit log to the default logs folder +on a linux installation with a log rotate size of 20MB run the command below.

    +
    +
    +
    $ couchbase-cli setting-audit -c 192.168.1.5:8091 --username Administrator \
+ --password password --enabled 1 --audit-log-rotate-size 20971520 \
+ --audit-log-path /opt/couchbase/var/lib/couchbase/logs
    +
    +

    To disable auditing run the following command.

    +
    +
    +
    $ couchbase-cli setting-audit -c 192.168.1.5:8091 --username Administrator \
+ --password password --enabled 1
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-admin-role-manage(1). +couchbase-ssl-manage(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-autofailover.dita b/content/cli/cbcli/couchbase-cli-setting-autofailover.dita deleted file mode 100644 index a73f951f8f..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-autofailover.dita +++ /dev/null @@ -1,155 +0,0 @@ - - - setting-autofailover - - Modifies auto-failover settings
    SYNTAX - - - couchbase-cli setting-autofailover [--cluster <url>] [--username <user>] - [--password <password>] [--enable-auto-failover <num>] - [--auto-failover-timeout <seconds>]
    DESCRIPTION - - -

    Auto-failover allows unresponsive servers to be failed over automatically by the -cluster manager. Data paritions in Couchbase are always served from a single -master node. As a result if a server is down in the cluster data from that -server will be not be available while that server is down. The server will -either need to be manually or automatically failed over in order to promote -replica data partitions on replica servers to active data partitions so that -they can be accessed by the application.

    -

    Since the administrator will not always be able to manually fail servers over -quickly enough to avoid application down time Couchbase provides an -auto-failover feature. This feature allows the cluster manager to automatically -fail over down nodes over and bring the cluster back to a healthy state as -quickly as possible.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---enable-auto-failover <num> -
    -
    -

    - Specifies whether or not autofailover is enabled. Set this option to "1" to - enable autofailover or "0" to disable autofailover. -

    -
    -     - -
    ---auto-failover-timeout <seconds> -
    -
    -

    - Specifies the auto-failover timeout. This is the amount of time a node must - be unresponsive before the cluster manager considers the node to be down and - fails it over. The minimum auto-failover timeout is 30 seconds in Couchbase - Community Edition and 5 seconds in Couchbase Enterprise Edition. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To enable autofailover with a 30 second auto-failover timeout run the following -command.

    - 
    $ couchbase-cli setting-autofailover -c 192.168.1.5 --username Administrator \
- --password password --enable-auto-failover 1 --auto-failover-timeout 30
    -

    To enable autofailover with a 120 second auto-failover timeout run the following -command.

    - 
    $ couchbase-cli setting-autofailover -c 192.168.1.5 --username Administrator \
- --password password --enable-auto-failover 1 --auto-failover-timeout 120
    -

    To disable autofailover run the following -command.

    - 
    $ couchbase-cli setting-autofailover -c 192.168.1.5 --username Administrator \
- --password password --enable-auto-failover 0
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-autofailover.html b/content/cli/cbcli/couchbase-cli-setting-autofailover.html new file mode 100644 index 0000000000..d6f17eceaa --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-autofailover.html @@ -0,0 +1,998 @@ + + + + + +couchbase-cli-setting-autofailover(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-autofailover [--cluster <url>] [--username <user>]
+          [--password <password>] [--enable-auto-failover <num>]
+          [--auto-failover-timeout <seconds>]
+          [--enable-failover-on-data-disk-issues <num>]
+          [--failover-data-disk-period <seconds>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Auto-failover allows unresponsive servers to be failed over automatically by the +cluster manager. Data partitions in Couchbase are always served from a single +master node. As a result if a server is down in the cluster data from that +server will be not be available while that server is down. The server will +either need to be manually or automatically failed over in order to promote +replica data partitions on replica servers to active data partitions so that +they can be accessed by the application.

    +

    Since the administrator will not always be able to manually fail servers over +quickly enough to avoid application down time Couchbase provides an +auto-failover feature. This feature allows the cluster manager to automatically +fail over down nodes over and bring the cluster back to a healthy state as +quickly as possible.

    +

    In Couchbase Server Enterprise Edition nodes can also be automatically failed +over when the Data Service reports sustained disk I/O failures.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--enable-auto-failover <num> +
    +
    +

    + Specifies whether or not autofailover is enabled. Set this option to "1" to + enable autofailover or "0" to disable autofailover. +

    +
    +
    +--auto-failover-timeout <seconds> +
    +
    +

    + Specifies the auto-failover timeout. This is the amount of time a node must + be unresponsive before the cluster manager considers the node to be down and + fails it over. The minimum auto-failover timeout is 30 seconds in Couchbase + Community Edition and 5 seconds in Couchbase Enterprise Edition. +

    +
    +
    +--max-failovers <num> +
    +
    +

    + Specifies the number of auto-failover events that will be handled before + requiring user intervention. A single event could be one node failing over or + an entire Server Group. The maximum allowed value is three. This feature is + only available in Couchbase Enterprise Edition. +

    +
    +
    +--enable-failover-of-server-groups <num> +
    +
    +

    + Specifies whether or not auto-failover can failover entire Server Groups. + This feature is only available in Couchbase Enterprise Edition. +

    +
    +
    +--enable-failover-on-data-disk-issues <num> +
    +
    +

    + Specifies whether or not autofailover on Data Service disk issues is enabled. + Set this option to "1" to enable failover on Data Service disk issue or + "0" to disable it. "--failover-data-disk-period" needs to be set at the + same time when enabling this option. This feature is only available in + Couchbase Enterprise Edition. +

    +
    +
    +--failover-data-disk-period <seconds> +
    +
    +

    + Specifies the failover data disk period. This is the period of time over + which the Data Service is checked for potential sustained Disk I/O + failures. The Data Service is checked every second for disk failures. If + 60% of the checks during that time period report disk failures, then the + node may be automatically failed over. + "--enable-failover-on-data-disk-issues" must be set when this option is + used. The failover data disk period ranges from 5 to 3600 seconds. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To enable autofailover with a 30 second auto-failover timeout run the following +command.

    +
    +
    +
    $ couchbase-cli setting-autofailover -c 192.168.1.5 --username Administrator \
+ --password password --enable-auto-failover 1 --auto-failover-timeout 30
    +
    +

    To enable autofailover with a 120 second auto-failover timeout run the following +command.

    +
    +
    +
    $ couchbase-cli setting-autofailover -c 192.168.1.5 --username Administrator \
+ --password password --enable-auto-failover 1 --auto-failover-timeout 120
    +
    +

    To enable autofailover with a 120 second auto-failover timeout and to enable +failover on Data Service Disk issue with a 10 second data disk period run the +following command.

    +
    +
    +
    $ couchbase-cli setting-autofailover -c 192.168.1.5 --username Administrator \
+ --password password --enable-auto-failover 1 --auto-failover-timeout 120 \
+ --enable-failover-on-data-disk-issues 1 --auto-failover-timeout 10
    +
    +

    To enable autofail with a 120 second auto-failover timeout and to enable failover +of a Server group and allow up to two auto failover event

    +
    +
    +
    $ couchbase-cli setting-autofailover -c 192.168.1.5 --username Administrator \
+ --password password --enable-auto-failover 1 --auto-failover-timeout 120 \
+ --enable-failover-of-server-groups 1 --max-failovers 2
    +
    +

    To disable autofailover run the following command.

    +
    +
    +
    $ couchbase-cli setting-autofailover -c 192.168.1.5 --username Administrator \
+ --password password --enable-auto-failover 0
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-failover(1). +couchbase-cli-recovery(1). +couchbase-cli-setting-alert(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-autoreprovision.dita b/content/cli/cbcli/couchbase-cli-setting-autoreprovision.dita deleted file mode 100644 index 7000dfc1ad..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-autoreprovision.dita +++ /dev/null @@ -1,164 +0,0 @@ - - - setting-autoreprovision - - Modifies auto-reprovision settings
    SYNTAX - - - couchbase-cli setting-autoreprovision [--cluster <url>] [--username <user>] - [--password <password>] [--enabled <1|0>] [--max-nodes <num>]
    DESCRIPTION - - -

    Auto-reprovisioning is used in order to prevent dataloss in ephemeral buckets -during failure scenarios when a node crashes and restarts quickly. Under this -scenario auto-reprovisioning ensures that an ephemeral bucket’s replica vbuckets -are prmoted to active state. This mechanism is not needed for Couchbase buckets -because a Couchbase buckets data is persisted to disk and can be loaded back -into memory after a node restarts. Ephemeral buckets on the other hand -completely lose their data when a node crashes and replicas must be relied upon -in order to prevent dataloss. If ephemeral buckets are not in use this setting -has no effect on the cluster.

    -

    Auto-reprovisioning can either be enabled or disabled. If you have ephemeral -buckets it is always recommended that auto-reprovisioning is enabled otherwise -the cluster will experience dataloss if a node crashes or is restarted. Users -can also specify the number of nodes that can be auto-reprovisioned before the -cluster is rebalanced. An auto-reprovision event occurs if a cluster has at -least one ephemeral bucket and a node crashes and restarts. If the number of -crashes and restarts exceed the maximum number of nodes that can be -auto-reprovisioned then the next crash and restart will result in dataloss for -all ephemeral buckets in the cluster.

    -

    When setting the max nodes parameter, note that specifying the max nodes to be -too high could result in cascading node failures. This can happen because when a -node fails and restarts the server load for the crashed node is distributed to -the rest of the servers in the cluster. This extra load could cause the one or -more of the remaining servers in the cluster to become overloaded and -unresponsive leading to more failures. On the other hand specifying max nodes to -be too small could lead to dataloss if there are many failures in the cluster at -the same time. How this variable is set depends on the cluster size, workload, -and configuration. However, it is always recommended that auto-reprovisioning is -at least enabled and that max nodes is set to at least 1.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---enabled <num> -
    -
    -

    - Specifies whether or not auto-reprovisioning is enabled. Set this option to - "1" to enable autofailover or "0" to disable autofailover. -

    -
    -     - -
    ---max-nodes <num> -
    -
    -

    - Specifies the maximum number of servers that can be auto-reprovisioned - before a rebalance must take place (which resets the count). This parameter - must always be set to a number greater than or equal to 1. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To enable auto-reprovisioning and allow up to three nodes be auto-reprovisioned -before a rebalance takes place run the following command.

    - 
    $ couchbase-cli setting-autoreprovision -c 192.168.1.5 -u Administrator \
- -p password --enabled 1 --max-nodes 3
    -

    To disable auto-reprovisioning run the following command.

    - 
    $ couchbase-cli setting-autoreprovision -c 192.168.1.5 -u Administrator \
- -p password --enabled 0
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-autoreprovision.html b/content/cli/cbcli/couchbase-cli-setting-autoreprovision.html new file mode 100644 index 0000000000..7ba2674030 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-autoreprovision.html @@ -0,0 +1,936 @@ + + + + + +couchbase-cli-setting-autoreprovision(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-autoreprovision [--cluster <url>] [--username <user>]
+          [--password <password>] [--enabled <1|0>] [--max-nodes <num>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Auto-reprovisioning is used in order to prevent data loss in ephemeral buckets +during failure scenarios when a node crashes and restarts quickly. Under this +scenario auto-reprovisioning ensures that an ephemeral bucket’s replica vBuckets +are promoted to active state. This mechanism is not needed for Couchbase buckets +because a Couchbase buckets data is persisted to disk and can be loaded back +into memory after a node restarts. Ephemeral buckets on the other hand +completely lose their data when a node crashes and replicas must be relied upon +in order to prevent data loss. If ephemeral buckets are not in use this setting +has no effect on the cluster.

    +

    Auto-reprovisioning can either be enabled or disabled. If you have ephemeral +buckets it is always recommended that auto-reprovisioning is enabled otherwise +the cluster will experience data loss if a node crashes or is restarted. Users +can also specify the number of nodes that can be auto-reprovisioned before the +cluster is rebalanced. An auto-reprovision event occurs if a cluster has at +least one ephemeral bucket and a node crashes and restarts. If the number of +crashes and restarts exceed the maximum number of nodes that can be +auto-reprovisioned then the next crash and restart will result in data loss for +all ephemeral buckets in the cluster.

    +

    When setting the max nodes parameter, note that specifying the max nodes to be +too high could result in cascading node failures. This can happen because when a +node fails and restarts the server load for the crashed node is distributed to +the rest of the servers in the cluster. This extra load could cause the one or +more of the remaining servers in the cluster to become overloaded and +unresponsive leading to more failures. On the other hand specifying max nodes to +be too small could lead to data loss if there are many failures in the cluster at +the same time. How this variable is set depends on the cluster size, workload, +and configuration. However, it is always recommended that auto-reprovisioning is +at least enabled and that max nodes is set to at least 1.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--enabled <num> +
    +
    +

    + Specifies whether or not auto-reprovisioning is enabled. Set this option to + "1" to enable autofailover or "0" to disable autofailover. +

    +
    +
    +--max-nodes <num> +
    +
    +

    + Specifies the maximum number of servers that can be auto-reprovisioned + before a rebalance must take place (which resets the count). This parameter + must always be set to a number greater than or equal to 1. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To enable auto-reprovisioning and allow up to three nodes be auto-reprovisioned +before a rebalance takes place run the following command.

    +
    +
    +
    $ couchbase-cli setting-autoreprovision -c 192.168.1.5 -u Administrator \
+ -p password --enabled 1 --max-nodes 3
    +
    +

    To disable auto-reprovisioning run the following command.

    +
    +
    +
    $ couchbase-cli setting-autoreprovision -c 192.168.1.5 -u Administrator \
+ -p password --enabled 0
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-autofailover(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-cluster.dita b/content/cli/cbcli/couchbase-cli-setting-cluster.dita deleted file mode 100644 index 8dcc852acb..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-cluster.dita +++ /dev/null @@ -1,212 +0,0 @@ - - - setting-cluster - - Modifies cluster settings
    SYNTAX - - - couchbase-cli setting-cluster [--cluster <url>] [--cluster-username <username>] - [--cluster-password <password>] [--cluster-port <port>] - [--cluster-ramsize <megabytes>] [--cluster-name <name>] - [--cluster-index-ramsize <megabytes>] [--cluster-fts-ramsize <megabytes>]
    DESCRIPTION - - -

    This command is used to modify cluster level settings. It allows users to change -the Couchbase root username and password, change the port that the cluster -manager listens on, and modify the data, index, and full-text service memory -quotas.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---cluster-username -
    -
    -

    - Specifies the new username for the Couchbase administrator user. -

    -
    -     - -
    ---cluster-password -
    -
    -

    - Specifies the new password for the Couchbase administrator user. -

    -
    -     - -
    ---cluster-ramsize -
    -
    -

    - Specifies the data services memory quota. This quota will be assigned to all - future nodes added to the cluster with the data service. -

    -
    -     - -
    ---cluster-fts-ramsize -
    -
    -

    - Sets the full-text service memory quota. This parameter is required if the - full-text service is specified in the --services option. This quota will be - assigned to all future nodes added to the cluster with the full-text - service. -

    -
    -     - -
    ---cluster-index-ramsize -
    -
    -

    - Sets the index service memory quota. This parameter is required if the index - service is specified in the --services option. This quota will be assigned - to all future nodes added to the cluster with the index service. -

    -
    -     - -
    ---cluster-name -
    -
    -

    - Sets the name for this cluster. Naming clusters is useful when you have - multiple Couchbase clusters in your deployment. -

    -
    -     - -
    ---cluster-port -
    -
    -

    - Specifies the port for the Couchbase Administration Console. Defaults to - port 8091. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To change the username and password of the Couchbase administrator user run the -following command.

    - 
    $ couchbase-cli setting-cluster -c 192.168.1.5:8091 --username cur_user \
- --password cur_password --cluster-username new_username \
- --cluster-password new_password
    -

    To change the port number that the cluster manager listens on and the cluster -name to "new_name" run the following command.

    - 
    $ couchbase-cli setting-cluster -c 192.168.1.5:8091 --username Administrator \
- --password password --cluster-port 5000 --cluster-name new_name
    -

    To change the memory quota of the data server to 2048 and the memory quota of -the index service to 4096 run the following command.

    - 
    $ couchbase-cli setting-cluster -c 192.168.1.5:8091 --username Administrator \
- --password password --cluster-ramsize 2048 --cluster-index-ramsize 4096
    -

    All of the parameters in this command can be specified at the same time. To -change the username and password of the Couchbase administrator user, change the -port number to 5000, change the cluster name to "new_name", change the memory -quota of the data service to 2048 and change the memory quota of the index -service to 4096 run the following command.

    - 
    $ couchbase-cli setting-cluster -c 192.168.1.5:8091 --username cur_user \
- --password cur_password --cluster-username new_username \
- --cluster-password new_password --cluster-port 5000 \
- --cluster-name new_name --cluster-ramsize 2048 --cluster-index-ramsize 4096
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-cluster.html b/content/cli/cbcli/couchbase-cli-setting-cluster.html new file mode 100644 index 0000000000..72bf2fbada --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-cluster.html @@ -0,0 +1,992 @@ + + + + + +couchbase-cli-setting-cluster(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-cluster [--cluster <url>] [--cluster-username <username>]
+          [--cluster-password <password>] [--cluster-port <port>]
+      [--cluster-ramsize <megabytes>] [--cluster-name <name>]
+      [--cluster-index-ramsize <megabytes>] [--cluster-fts-ramsize <megabytes>]
+      [--cluster-eventing-ramsize <megabytes>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command is used to modify cluster level settings. It allows users to change +the Couchbase root username and password, change the port that the cluster +manager listens on, and modify the data, index, and full-text service memory +quotas.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--cluster-username +
    +
    +

    + Specifies the new username for the Couchbase administrator user. +

    +
    +
    +--cluster-password +
    +
    +

    + Specifies the new password for the Couchbase administrator user. +

    +
    +
    +--cluster-ramsize +
    +
    +

    + Specifies the data services memory quota. This quota will be assigned to all + future nodes added to the cluster with the data service. +

    +
    +
    +--cluster-fts-ramsize +
    +
    +

    + Sets the full-text service memory quota. This parameter is required if the + full-text service is specified in the --services option. This quota will be + assigned to all future nodes added to the cluster with the full-text + service. +

    +
    +
    +--cluster-index-ramsize +
    +
    +

    + Sets the index service memory quota. This parameter is required if the index + service is specified in the --services option. This quota will be assigned + to all future nodes added to the cluster with the index service. +

    +
    +
    +--cluster-eventing-ramsize +
    +
    +

    + Sets the Eventing service memory quota (in MB). This parameter is required + if the Eventing service is specified in the --services option. This quota + will be assigned to all future nodes added to the cluster with the eventing + service. +

    +
    +
    +--cluster-name +
    +
    +

    + Sets the name for this cluster. Naming clusters is useful when you have + multiple Couchbase clusters in your deployment. +

    +
    +
    +--cluster-port +
    +
    +

    + Specifies the port for the Couchbase Administration Console. Defaults to + port 8091. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To change the username and password of the Couchbase administrator user run the +following command.

    +
    +
    +
    $ couchbase-cli setting-cluster -c 192.168.1.5:8091 --username cur_user \
+ --password cur_password --cluster-username new_username \
+ --cluster-password new_password
    +
    +

    To change the port number that the cluster manager listens on and the cluster +name to "new_name" run the following command.

    +
    +
    +
    $ couchbase-cli setting-cluster -c 192.168.1.5:8091 --username Administrator \
+ --password password --cluster-port 5000 --cluster-name new_name
    +
    +

    To change the memory quota of the data server to 2048 and the memory quota of +the index service to 4096 run the following command.

    +
    +
    +
    $ couchbase-cli setting-cluster -c 192.168.1.5:8091 --username Administrator \
+ --password password --cluster-ramsize 2048 --cluster-index-ramsize 4096
    +
    +

    All of the parameters in this command can be specified at the same time. To +change the username and password of the Couchbase administrator user, change the +port number to 5000, change the cluster name to "new_name", change the memory +quota of the data service to 2048 and change the memory quota of the index +service to 4096 run the following command.

    +
    +
    +
    $ couchbase-cli setting-cluster -c 192.168.1.5:8091 --username cur_user \
+ --password cur_password --cluster-username new_username \
+ --cluster-password new_password --cluster-port 5000 \
+ --cluster-name new_name --cluster-ramsize 2048 --cluster-index-ramsize 4096
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-cluster-init(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-compaction.dita b/content/cli/cbcli/couchbase-cli-setting-compaction.dita deleted file mode 100644 index b141d58836..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-compaction.dita +++ /dev/null @@ -1,388 +0,0 @@ - - - setting-compaction - - Modifies compaction settings
    SYNTAX - - - couchbase-cli setting-compaction [--cluster <url>] [--username <user>] - [--password <password>] [--compaction-db-percentage <num>] - [--compaction-db-size <megabytes>] [--compaction-view-percentage <num>] - [--compaction-view-size <megabytes>] [--compaction-period-from <HH:MM>] - [--compaction-period-to <HH:MM>] [--enable-compaction-abort <num>] - [--enable-compaction-parallel <num>] [--metadata-purge-interval <num>] - [--gsi-compaction-mode <mode>] [--compaction-gsi-percentage <percent>] - [--compaction-gsi-interval <list_of_days>] - [--compaction-gsi-period-from <HH:MM>] - [--compaction-gsi-period-to <HH:MM>] [--enable-gsi-compaction-abort <1|0>]
    DESCRIPTION - - -

    This command sets cluster-wide compaction settings for the views and data -service.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---compaction-db-percentage <num> -
    -
    -

    - Compacts database files once the framentation percentage is greater than - the value set for this option. The value must be between 2 and 100. -

    -
    -     - -
    ---compaction-db-size <megabytes> -
    -
    -

    - Compacts the database files once the file fragmentation (in MB) is greater - than the value of this option. This option must be set to a value - greater than 1. -

    -
    -     - -
    ---compaction-view-percentage <num> -
    -
    -

    - Compacts view files once the framentation percentage is greater than the - value set for this option. The value must be between 2 and 100. -

    -
    -     -
    -

    --compaction-view-size: <megabytes>: - Compacts the view files once the file fragmentation (in MB) is greater than - the value of this option. This option must be set to a value greater than 1.

    -
    - -
    ---compaction-period-from <HH:MM> -
    -
    -

    - This option is unsed in conjunction with the --compaction-period-to option - and is used to specify a time period where compaction is allowed to run. You - could for example specify that compaction should only run between midnight - and 5AM each day by setting the compaction from period to "00:00" and the - compaction to period to "5:00". When setting the value for this option you - must use the format HH:MM when HH corresponds to the hour and MM corresponds - to the minute. If this option is not specified then the compaction will run - at any time of the day. This option only affects view and database file - compaction. -

    -
    -     - -
    ---compaction-period-to <HH:MM> -
    -
    -

    - This option is unsed in conjunction with the --compaction-period-from option - and is used to specify a time period where compaction is allowed to run. You - could for example specify that compaction should only run between midnight - and 5AM each day by setting the compaction from period to "00:00" and the - compaction to period to "5:00". When setting the value for this option you - must use the format HH:MM when HH corresponds to the hour and MM corresponds - to the minute. If this option is not specified then the compaction will run - at any time of the day. This option only affects view and database file - compaction. -

    -
    -     - -
    ---enable-compaction-abort <num> -
    -
    -

    - If a compaction from period and compaction to period are specified then this - flag tells the server how to respond if a compaction starts during the - allowed compaction interval and is still running once after the allowed - interval has ended. If this option is set to "1" then the compaction will - be aborted. If it is set to "0" then the compaction will be allowed to - complete. By default this option is set to "0". -

    -
    -     - -
    ---enable-compaction-parallel <num> -
    -
    -

    - Specifies whether view and database file compaction can run at the same - time. Compactions can be disk intensive operations so it may be beneficial - to only allow one type of compaction to run at a time. To allow parallel - compactions set the value of this option to "1". To disable parallel - compaction set the value fo this option to "0". By default this option is - set to "0". -

    -
    -     - -
    ---metadata-purge-interval <days> -
    -
    -

    - Couchbase persists deletes to disk because these deletes may need to be - replicated in the future during intra-cluster replication as well as during - cross datacenter replication. Couchbase cannot however keep these deletes - forever because they will cause the database disk size to increase - infinitely over time. To combat this issue Couchbase purges old deletes from - disk periodically. This flag allow the user to set this interval. By default - the purge interval is set to 7 days. This means that we purge deletes from - disk that are more than 7 days old. The value of this option must be between - 0.04 (1 hour) and 60 (days). -

    -
    -     - -
    ---gsi-compaction-mode <mode> -
    -
    -

    - Specifies the strategy for compaction in GSI Indexes. This option may be set - to either append or circular. The append compaction strategy works by - creating a new index file, moving the active data to the new index file, and - then removing the old index file. This strategy will cause increased disk - usage during compaction, but will cause the new index file to be smaller - than the old one and as a result will free up disk space. The circular - compaction strategy uses the same file, but moves data around in the file to - create contigious free space which can be reused. This strategy uses no new - space during compaction, but the resulting file size will not be decreased - since the compacted file will result in a section of active data and free - space. -

    -
    -     - -
    ---compaction-gsi-percentage <percent> -
    -
    -

    - Specifies that gsi compaction should be started when the fragmentation in an - index file has exceeded this percentage. This parameter only applies if the - append compaction mode is used. -

    -
    -     - -
    ---compaction-gsi-interval <list_of_days> -
    -
    -

    - Specifies that gsi compaction should only run on the specified days. This - option takes a comma separated list of days where the name of the day is - capitalized. Accepted values are Monday, Tuesday, Wednesday, Thursday, - Friday, Saturday, and Sunday. If you only want compaction to run on Monday - and Tuesday then the value of this option should be set to "Monday,Tuesday". - This parameter only applies if circular compaction mode is used. -

    -
    -     - -
    ---compaction-gsi-period-from <HH:MM> -
    -
    -

    - This option is unsed in conjunction with the --compaction-gsi-period-to - option and is used to specify a time period where gsi compaction is allowed - to run. You could for example specify that gsi compaction should only run - between midnight and 5AM each day by setting the gsi compaction from period - to "00:00" and the gsi compaction to period to "5:00". When setting the - value for this option you must use the format HH:MM when HH corresponds to - the hour and MM corresponds to the minute. If this option is not specified - then compaction will run at any time of the day. This parameter only - applies if circular compaction mode is used. -

    -
    -     - -
    ---compaction-gsi-period-to <HH:MM> -
    -
    -

    - This option is unsed in conjunction with the --compaction-gsi-period-from - option and is used to specify a time period where gsi compaction is allowed - to run. You could for example specify that gsi compaction should only run - between midnight and 5AM each day by setting the gsi compaction from period - to "00:00" and the gsi compaction to period to "5:00". When setting the - value for this option you must use the format HH:MM when HH corresponds to - the hour and MM corresponds to the minute. If this option is not specified - then gsi compaction will run at any time of the day. This parameter only - applies if circular compaction mode is used. -

    -
    -     - -
    ---enable-gsi-compaction-abort <1|0> -
    -
    -

    - If a gsi compaction from period and gsi compaction to period are specified - then this flag tells the server how to respond if a compaction starts during - the allowed gsi compaction interval and is still running after the allowed - interval has ended. If this option is set to "1" then the gsi compaction - will be aborted. If it is set to "0" then the gsi compaction will be allowed - to complete. By default this option is set to "0". This parameter only - applies if circular compaction mode is used. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    If we want to set our view and database compaction percentage thresholds to 30% -each, but also wanted to ensure that our fragementation didn’t grow above 1GB -we would run the following command

    - 
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
- --password password --compaction-view-size 1024 --compaction-db-size 1024 \
- --compaction-view-percentage 30 --compaction-db-percentage 30
    -

    If we want to have the same settings as above, but we wanted compaction to only -run at night so that we didn’t run the risk of compaction affecting normal -application traffic we would run the following command. Note that in this -example we will assume our night time period is midnight to 6AM. We will also -enable compaction aborts so that we can ensure compaction is never running -outside of this time window.

    - 
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
- --password password --compaction-view-size 1024 --compaction-db-size 1024 \
- --compaction-view-percentage 30 --compaction-db-percentage 30 \
- --compaction-period-from 00:00 --compaction-period-to 6:00 \
- --enable-compaction-abort 1
    -

    If we don’t mind when compaction runs and we have the disk overhead to run both -view and database compaction at the same time then we might set up comaction -with the settings in the first example, but also enable parallel compactions. -This can be done by running the command below.

    - 
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
- --password password --compaction-view-size 1024 --compaction-db-size 1024 \
- --compaction-view-percentage 30 --compaction-db-percentage 30 \
- --enable-compaction-parallel
    -

    If your application heavily uses expirations or you create and delete a lot of -documents quickly then you might want to shorten your metadata purge interval -in order to ensure that you don’t use too much disk space. If we want our -compactions to run when the fragmentation is 30% or 1GB and we want to change -the metadata purge interval to 2 days then we would run the following command.

    - 
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
- --password password --compaction-view-size 1024 --compaction-db-size 1024 \
- --compaction-view-percentage 30 --compaction-db-percentage 30 \
- --meta-data-purge-interval 2
    -

    If you need to change the GSI index compaction settings to use the append -compaction mode and want gsi compaction only to happen once your file is 50% -fragmented specify the following command.

    - 
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
- --password password --gsi-compaction-mode append \
- --compaction-gsi-percentage 50
    -

    If you want to change the GSI index compaction settings to use the circular -compaction mode and want gsi compaction only to happen on Tuesdays and Thursdays -between midnight and 3AM and don’t want gsi compaction running outside of those -time windows even if the compaction started at a valid time specify the -following command.

    - 
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
- --password password --gsi-compaction-mode circular \
- --compaction-gsi-interval Tuesday,Thursday \
- --compaction-gsi-period-from 00:00 \
- --compaction-gsi-period-to 3:00 --enable-gsi-compaction-abort 1
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-compaction.html b/content/cli/cbcli/couchbase-cli-setting-compaction.html new file mode 100644 index 0000000000..58130ffbd3 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-compaction.html @@ -0,0 +1,1147 @@ + + + + + +couchbase-cli-setting-compaction(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-compaction [--cluster <url>] [--username <user>]
+          [--password <password>] [--compaction-db-percentage <num>]
+          [--compaction-db-size <megabytes>] [--compaction-view-percentage <num>]
+          [--compaction-view-size <megabytes>] [--compaction-period-from <HH:MM>]
+          [--compaction-period-to <HH:MM>] [--enable-compaction-abort <num>]
+          [--enable-compaction-parallel <num>] [--metadata-purge-interval <num>]
+          [--gsi-compaction-mode <mode>] [--compaction-gsi-percentage <percent>]
+          [--compaction-gsi-interval <list_of_days>]
+          [--compaction-gsi-period-from <HH:MM>]
+          [--compaction-gsi-period-to <HH:MM>] [--enable-gsi-compaction-abort <1|0>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command sets cluster-wide compaction settings for the views and data +service.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--compaction-db-percentage <num> +
    +
    +

    + Compacts database files once the fragmentation percentage is greater than + the value set for this option. The value must be between 2 and 100. +

    +
    +
    +--compaction-db-size <megabytes> +
    +
    +

    + Compacts the database files once the file fragmentation (in MB) is greater + than the value of this option. This option must be set to a value + greater than 1. +

    +
    +
    +--compaction-view-percentage <num> +
    +
    +

    + Compacts view files once the fragmentation percentage is greater than the + value set for this option. The value must be between 2 and 100. +

    +
    +
    +

    --compaction-view-size: <megabytes>: + Compacts the view files once the file fragmentation (in MB) is greater than + the value of this option. This option must be set to a value greater than 1.

    +
    +
    +--compaction-period-from <HH:MM> +
    +
    +

    + This option is unison in conjunction with the --compaction-period-to option + and is used to specify a time period where compaction is allowed to run. You + could for example specify that compaction should only run between midnight + and 5AM each day by setting the compaction from period to "00:00" and the + compaction to period to "5:00". When setting the value for this option you + must use the format HH:MM when HH corresponds to the hour and MM corresponds + to the minute. If this option is not specified then the compaction will run + at any time of the day. This option only affects view and database file + compaction. +

    +
    +
    +--compaction-period-to <HH:MM> +
    +
    +

    + This option is unison in conjunction with the --compaction-period-from option + and is used to specify a time period where compaction is allowed to run. You + could for example specify that compaction should only run between midnight + and 5AM each day by setting the compaction from period to "00:00" and the + compaction to period to "5:00". When setting the value for this option you + must use the format HH:MM when HH corresponds to the hour and MM corresponds + to the minute. If this option is not specified then the compaction will run + at any time of the day. This option only affects view and database file + compaction. +

    +
    +
    +--enable-compaction-abort <num> +
    +
    +

    + If a compaction from period and compaction to period are specified then this + flag tells the server how to respond if a compaction starts during the + allowed compaction interval and is still running once after the allowed + interval has ended. If this option is set to "1" then the compaction will + be aborted. If it is set to "0" then the compaction will be allowed to + complete. By default this option is set to "0". +

    +
    +
    +--enable-compaction-parallel <num> +
    +
    +

    + Specifies whether view and database file compaction can run at the same + time. Compaction can be disk intensive operations so it may be beneficial + to only allow one type of compaction to run at a time. To allow parallel + compaction set the value of this option to "1". To disable parallel + compaction set the value of this option to "0". By default this option is + set to "0". +

    +
    +
    +--metadata-purge-interval <days> +
    +
    +

    + Couchbase persists deletes to disk because these deletes may need to be + replicated in the future during intra-cluster replication as well as during + Cross Data Center Replication. Couchbase cannot however keep these deletes + forever because they will cause the database disk size to increase + infinitely over time. To combat this issue Couchbase purges old deletes from + disk periodically. This flag allow the user to set this interval. By default + the purge interval is set to 7 days. This means that we purge deletes from + disk that are more than 7 days old. The value of this option must be between + 0.04 (1 hour) and 60 (days). +

    +
    +
    +--gsi-compaction-mode <mode> +
    +
    +

    + Specifies the strategy for compaction in GSI Indexes. This option may be set + to either append or circular. The append compaction strategy works by + creating a new index file, moving the active data to the new index file, and + then removing the old index file. This strategy will cause increased disk + usage during compaction, but will cause the new index file to be smaller + than the old one and as a result will free up disk space. The circular + compaction strategy will append data at first until the index is 65% + fragmented. At which point it will start to write data over old blocks in the + file that are no longer being used. A full compaction will be triggered once a + day on the day set via --compaction-gsi-interval. +

    +
    +
    +--compaction-gsi-percentage <percent> +
    +
    +

    + Specifies that GSI compaction should be started when the fragmentation in an + index file has exceeded this percentage. This parameter only applies if the + append compaction mode is used. +

    +
    +
    +--compaction-gsi-interval <list_of_days> +
    +
    +

    + Specifies that GSI compaction should only run on the specified days. This + option takes a comma separated list of days where the name of the day is + capitalized. Accepted values are Monday, Tuesday, Wednesday, Thursday, + Friday, Saturday, and Sunday. If you only want compaction to run on Monday + and Tuesday then the value of this option should be set to "Monday,Tuesday". + This parameter only applies if circular compaction mode is used. +

    +
    +
    +--compaction-gsi-period-from <HH:MM> +
    +
    +

    + This option is unison in conjunction with the --compaction-gsi-period-to + option and is used to specify a time period where GSI compaction is allowed + to run. You could for example specify that GSI compaction should only run + between midnight and 5AM each day by setting the GSI compaction from period + to "00:00" and the GSI compaction to period to "5:00". When setting the + value for this option you must use the format HH:MM when HH corresponds to + the hour and MM corresponds to the minute. If this option is not specified + then compaction will run at any time of the day. This parameter only + applies if circular compaction mode is used. +

    +
    +
    +--compaction-gsi-period-to <HH:MM> +
    +
    +

    + This option is unison in conjunction with the --compaction-gsi-period-from + option and is used to specify a time period where GSI compaction is allowed + to run. You could for example specify that GSI compaction should only run + between midnight and 5AM each day by setting the GSI compaction from period + to "00:00" and the GSI compaction to period to "5:00". When setting the + value for this option you must use the format HH:MM when HH corresponds to + the hour and MM corresponds to the minute. If this option is not specified + then GSI compaction will run at any time of the day. This parameter only + applies if circular compaction mode is used. +

    +
    +
    +--enable-gsi-compaction-abort <1|0> +
    +
    +

    + If a GSI compaction from period and GSI compaction to period are specified + then this flag tells the server how to respond if a compaction starts during + the allowed GSI compaction interval and is still running after the allowed + interval has ended. If this option is set to "1" then the GSI compaction + will be aborted. If it is set to "0" then the GSI compaction will be allowed + to complete. By default this option is set to "0". This parameter only + applies if circular compaction mode is used. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    If we want to set our view and database compaction percentage thresholds to 30% +each, but also wanted to ensure that our fragmentation didn’t grow above 1GB +we would run the following command

    +
    +
    +
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
+ --password password --compaction-view-size 1024 --compaction-db-size 1024 \
+ --compaction-view-percentage 30 --compaction-db-percentage 30
    +
    +

    If we want to have the same settings as above, but we wanted compaction to only +run at night so that we didn’t run the risk of compaction affecting normal +application traffic we would run the following command. Note that in this +example we will assume our night time period is midnight to 6AM. We will also +enable compaction aborts so that we can ensure compaction is never running +outside of this time window.

    +
    +
    +
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
+ --password password --compaction-view-size 1024 --compaction-db-size 1024 \
+ --compaction-view-percentage 30 --compaction-db-percentage 30 \
+ --compaction-period-from 00:00 --compaction-period-to 6:00 \
+ --enable-compaction-abort 1
    +
    +

    If we don’t mind when compaction runs and we have the disk overhead to run both +view and database compaction at the same time then we might set up compaction +with the settings in the first example, but also enable parallel compaction. +This can be done by running the command below.

    +
    +
    +
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
+ --password password --compaction-view-size 1024 --compaction-db-size 1024 \
+ --compaction-view-percentage 30 --compaction-db-percentage 30 \
+ --enable-compaction-parallel
    +
    +

    If your application heavily uses expirations or you create and delete a lot of +documents quickly then you might want to shorten your metadata purge interval +in order to ensure that you don’t use too much disk space. If we want our +compaction to run when the fragmentation is 30% or 1GB and we want to change +the metadata purge interval to 2 days then we would run the following command.

    +
    +
    +
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
+ --password password --compaction-view-size 1024 --compaction-db-size 1024 \
+ --compaction-view-percentage 30 --compaction-db-percentage 30 \
+ --meta-data-purge-interval 2
    +
    +

    If you need to change the GSI index compaction settings to use the append +compaction mode and want GSI compaction only to happen once your file is 50% +fragmented specify the following command.

    +
    +
    +
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
+ --password password --gsi-compaction-mode append \
+ --compaction-gsi-percentage 50
    +
    +

    If you want to change the GSI index compaction settings to use the circular +compaction mode and want GSI compaction only to happen on Tuesdays and Thursdays +between midnight and 3AM and don’t want GSI compaction running outside of those +time windows even if the compaction started at a valid time specify the +following command.

    +
    +
    +
    $ couchbase-cli setting-compaction -c 192.168.1.5 --username Administrator \
+ --password password --gsi-compaction-mode circular \
+ --compaction-gsi-interval Tuesday,Thursday \
+ --compaction-gsi-period-from 00:00 \
+ --compaction-gsi-period-to 3:00 --enable-gsi-compaction-abort 1
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-bucket-compact(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-index.dita b/content/cli/cbcli/couchbase-cli-setting-index.dita deleted file mode 100644 index ac4416af5e..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-index.dita +++ /dev/null @@ -1,198 +0,0 @@ - - - setting-index - - Modifies index settings
    SYNTAX - - - couchbase-cli setting-index [--cluster <url>] [--username <user>] - [--password <password>] [--index-max-rollback-points <num>] - [--index-stable-snapshot-interval <seconds>] - [--index-memory-snapshot-interval <milliseconds>] - [--index-storage-setting <type>] [--index-threads <num>] - [--index-log-level <level>]
    DESCRIPTION - - -

    This command sets various advanced settings for the index service.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---index-max-rollback-points <num> -
    -
    -

    - The maximum number of rollback points. The value of this option must be - greater than or equal to 1. -

    -
    -     - -
    ---index-stable-snapshot-interval <seconds> -
    -
    -

    - Specifies the frequency of persisted snapshots for recovery in seconds. This - means that in the event of a failover this is the farthest back we may have - to rebuild the index from. This value of this parameter must be greater than -

    -
    1. -

      -If this options is not specified it defaults to 30000 seconds (8 hours - 20 minutes). -

      -
    -
    -     - -
    ---index-memory-snapshot-interval <milliseconds> -
    -
    -

    - Specifies the frequency of in-memory snapshots in milliseconds. This - determines the earliest possibility of a scan seeing a given KV mutation. - This value of this parameter must be greater than 1. This parameter defaults - to 200 if it is not specified. -

    -
    -     - -
    ---index-storage-setting <type> -
    -
    -

    - Sets the index storage mode for all indexes in this cluster. This parameter - may only be changed if there are no nodes running the index service. To - specify the disk based forestdb index backend set this parameter to - "default". To set the storage mode to use memory optimized indexes set this - parameter to "memopt". -

    -
    -     - -
    ---index-threads <num> -
    -
    -

    - Sets the number of CPUs that can be used by the indexer. The value of this - option must be between 0 and 1024. If this option is not set then it - default to 4. -

    -
    -     - -
    ---index-log-level <level> -
    -
    -

    - Sets the log level for the index service. Valid log levels include "debug", - "silent", "fatal", "error", "warn", "info", "verbose", "timing", and - "trace". -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To set the max index rollback points to 8, the index stable snapshot interval to -40000 seconds, the index memory snapshot interval to 150 milliseconds, the index -thread count to 8 and in thedex log level to info run the following command.

    - 
       $ couchbase-cli setting-index -c 127.0.0.1:8091 -u Administrator \
-    -p password --index-max-rollback-points 8 --index-threads 8 \
---index-log-level info --index-stable-snapshot-interval 40000 \
---index-memory-snapshot-interval 150 \
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-index.html b/content/cli/cbcli/couchbase-cli-setting-index.html new file mode 100644 index 0000000000..87dbebaf67 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-index.html @@ -0,0 +1,961 @@ + + + + + +couchbase-cli-setting-index(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-index [--cluster <url>] [--username <user>]
+          [--password <password>] [--index-max-rollback-points <num>]
+          [--index-stable-snapshot-interval <seconds>]
+          [--index-memory-snapshot-interval <milliseconds>]
+          [--index-storage-setting <type>] [--index-threads <num>]
+          [--index-log-level <level>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command sets various advanced settings for the index service.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--index-max-rollback-points <num> +
    +
    +

    + The maximum number of rollback points. The value of this option must be + greater than or equal to 1. +

    +
    +
    +--index-stable-snapshot-interval <seconds> +
    +
    +

    + Specifies the frequency of persisted snapshots for recovery in seconds. This + means that in the event of a failover this is the farthest back we may have + to rebuild the index from. This value of this parameter must be greater than +

    +
      +
    1. +

      +If this options is not specified it defaults to 30000 seconds (8 hours + 20 minutes). +

      +
      2. +
    +
    +
    +--index-memory-snapshot-interval <milliseconds> +
    +
    +

    + Specifies the frequency of in-memory snapshots in milliseconds. This + determines the earliest possibility of a scan seeing a given KV mutation. + This value of this parameter must be greater than 1. This parameter defaults + to 200 if it is not specified. +

    +
    +
    +--index-storage-setting <type> +
    +
    +

    + Sets the index storage mode for all indexes in this cluster. This parameter + may only be changed if there are no nodes running the index service. To + specify the disk based forestdb index backend set this parameter to + "default". To set the storage mode to use memory optimized indexes set this + parameter to "memopt". +

    +
    +
    +--index-threads <num> +
    +
    +

    + Sets the number of CPUs that can be used by the indexer. The value of this + option must be between 0 and 1024. If this option is not set then it + default to 4. +

    +
    +
    +--index-log-level <level> +
    +
    +

    + Sets the log level for the index service. Valid log levels include "debug", + "silent", "fatal", "error", "warn", "info", "verbose", "timing", and + "trace". +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To set the max index rollback points to 8, the index stable snapshot interval to +40000 seconds, the index memory snapshot interval to 150 milliseconds, the index +thread count to 8 and in the index log level to info run the following command.

    +
    +
    +
       $ couchbase-cli setting-index -c 127.0.0.1:8091 -u Administrator \
+    -p password --index-max-rollback-points 8 --index-threads 8 \
+--index-log-level info --index-stable-snapshot-interval 40000 \
+--index-memory-snapshot-interval 150 \
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-cluster(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-ldap.dita b/content/cli/cbcli/couchbase-cli-setting-ldap.dita deleted file mode 100644 index 3c00e50f1b..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-ldap.dita +++ /dev/null @@ -1,174 +0,0 @@ - - - setting-ldap - - Modifies LDAP settings
    SYNTAX - - - couchbase-cli setting-ldap [--cluster <url>] [--username <user>] - [--password <password>] [--ldap-admins <admins_list>] - [--ldap-roadmins <ro_admin_list>] [--ldap-default <default>] - [--ldap-enable <num>]
    DESCRIPTION - - -

    This command allows users to enable LDAP on their cluster as well as add -administrator and read-only administrator LDAP users to their cluster.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---ldap-admins <admins_list> -
    -
    -

    - A list of LDAP users who should have administrator privileges on this - cluster. This list should be comma separated if multiple users are being - added. -

    -
    -     - -
    ---ldap-roadmins <ro_admin_list> -
    -
    -

    - A list of LDAP users who should have read-only administrator privileges on - this cluster. This list should be comma separated if multiple users are - being added. -

    -
    -     - -
    ---ldap-enabled <num> -
    -
    -

    - Enables or disables LDAP on this cluster. To enable LDAP set this option to - "1". To disable LDAP set this parameter to "0". -

    -
    -     - -
    ---ldap-default <default> -
    -
    -

    - Specifies the default role for LDAP users who have not been explicitly been - added to the cluster. This option may be set to "admins", "ro_admins", or - "none". If this option is set to "admins" then all LDAP users not explicitly - add to this cluster have administrator privileges. If this option is set to - "roadmins" then all LDAP users not explicitly add to this cluster have - read-only administrator privileges. If this option is set to "none" then all - LDAP users not explicitly add to this cluster will have no access. This - option default to "none". -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To enable LDAP on a cluster run the following command.

    - 
    $ couchbase-cli setting-ldap -c 192.168.1.5 --username Administrator \
- --password password --enable-ldap 1
    -

    To add users alice and barry as Administrators and users clair and daniel as -Read-Only Administrators run the following command.

    - 
    $ couchbase-cli setting-ldap -c 192.168.1.5 --username Administrator \
- --password password --enable-ldap 1 --ldap-admins alice,barry \
- --ldap-ro-admins clair,daniel
    -

    To enable LDAP and allow all LDAP users to have Amdinistrator access run the -following command.

    - 
    $ couchbase-cli setting-ldap -c 192.168.1.5 --username Administrator \
- --password password --enable-ldap 1 --ldap-default admins
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-ldap.html b/content/cli/cbcli/couchbase-cli-setting-ldap.html new file mode 100644 index 0000000000..ae72def010 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-ldap.html @@ -0,0 +1,946 @@ + + + + + +couchbase-cli-setting-ldap(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-ldap [--cluster <url>] [--username <user>]
+          [--password <password>] [--ldap-admins <admins_list>]
+          [--ldap-roadmins <ro_admin_list>] [--ldap-default <default>]
+          [--ldap-enable <num>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command allows users to enable LDAP on their cluster as well as add +administrator and read-only administrator LDAP users to their cluster.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--ldap-admins <admins_list> +
    +
    +

    + A list of LDAP users who should have administrator privileges on this + cluster. This list should be comma separated if multiple users are being + added. +

    +
    +
    +--ldap-roadmins <ro_admin_list> +
    +
    +

    + A list of LDAP users who should have read-only administrator privileges on + this cluster. This list should be comma separated if multiple users are + being added. +

    +
    +
    +--ldap-enabled <num> +
    +
    +

    + Enables or disables LDAP on this cluster. To enable LDAP set this option to + "1". To disable LDAP set this parameter to "0". +

    +
    +
    +--ldap-default <default> +
    +
    +

    + Specifies the default role for LDAP users who have not been explicitly been + added to the cluster. This option may be set to "admins", "ro_admins", or + "none". If this option is set to "admins" then all LDAP users not explicitly + add to this cluster have administrator privileges. If this option is set to + "roadmins" then all LDAP users not explicitly add to this cluster have + read-only administrator privileges. If this option is set to "none" then all + LDAP users not explicitly add to this cluster will have no access. This + option default to "none". +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To enable LDAP on a cluster run the following command.

    +
    +
    +
    $ couchbase-cli setting-ldap -c 192.168.1.5 --username Administrator \
+ --password password --enable-ldap 1
    +
    +

    To add users alice and barry as Administrators and users clair and daniel as +Read-Only Administrators run the following command.

    +
    +
    +
    $ couchbase-cli setting-ldap -c 192.168.1.5 --username Administrator \
+ --password password --enable-ldap 1 --ldap-admins alice,barry \
+ --ldap-ro-admins clair,daniel
    +
    +

    To enable LDAP and allow all LDAP users to have Amdinistrator access run the +following command.

    +
    +
    +
    $ couchbase-cli setting-ldap -c 192.168.1.5 --username Administrator \
+ --password password --enable-ldap 1 --ldap-default admins
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-admin-role-manage(1). +couchbase-cli-user-manage(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-master-password.html b/content/cli/cbcli/couchbase-cli-setting-master-password.html new file mode 100644 index 0000000000..2cf1e36a61 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-master-password.html @@ -0,0 +1,945 @@ + + + + + +couchbase-cli-setting-master-password(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-master-password [--cluster <url>] [--username <user>]
+          [--password <password>] [--new-password <password>] [--rotate-password]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Couchbase Server Enterprise Edition has a "Secret Management" feature, which +allows users to securely encrypt passwords and other sensitive configuration +information that is stored on disk. These secrets must be stored in a secure +way; and access must be controlled, to reduce the risk of accidental exposure. +By using Secret Management in Couchbase Server, secrets are written to disk +in encrypted format. To decrypt these secrets, Couchbase requires entry of a +"master password", which is supplied by the user during server startup. This +master password can be passed to the server using the couchbase-cli +master-password command.

    +

    By default, the Secret Management feature is disabled. To enable the feature, +you must first set the master password. Once a master password is set, the +user is required to enter it when the server starts up. This can be done by +setting the environment variable CB_MASTER_PASSWORD=<password> during server +startup. Alternatively, you can set the environment variable +CB_WAIT_FOR_MASTER_PASSWORD=true, and then enter the master password using the +couchbase-cli master-password command.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--rotate-data-key +
    +
    +

    + Secrets are encrypted using a data key file, which is a unique key that is + stored on disk for each server. To open this file, the master password is + used to generate a key which decrypts the contents of the data key file. + The contents of the decrypted data key file can then be used to decrypt + secrets. Some users may want to generate a new data key file periodically, + to increase security. This option is used to generate a new data key file. +

    +
    +
    +--new-password +
    +
    +

    + Sets a new master password for the server specified. The user may specify + this password on the command line, or through non-echoed stdin. To specify + the password through non-echoed stdin, do not provide a value for this + option. The user will then be prompted to enter the password. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To use the Secret Management feature, the first thing you need to do is set a +password on each node of the cluster. To do this, install and start Couchbase, +but don’t go through the setup process or initialize the cluster. Once +Couchbase has started, run the following command to set the master password +for your server.

    +
    +
    +
    $ couchbase-cli setting-master-password -c 127.0.0.1 -u Administrator \
+  -p password --new-password password
    +
    +

    Once the master password has been set, you need to set the server environment +variable CB_WAIT_FOR_MASTER_PASSWORD=true. You can do this by running the +command below, or by setting the variable in your .bashrc file.

    +
    +
    +
    $ export CB_WAIT_FOR_MASTER_PASSWORD=true
    +
    +

    This environment variable will cause Couchbase to wait for the master password +before starting up. Once it is set, you need to restart your cluster. Upon +restarting the cluster you will notice that the server doesn’t fully start. +This is because it is waiting for you to enter the master password. You can do +this by running the command below. The master-password subcommand has to be +run locally on the node that is waiting for the master password.

    +
    +
    +
    $ couchbase-cli master-password --send-password password
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-master-password(1). +couchbase-cli-cluster-init(1). +couchbase-cli-server-add(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-notification.dita b/content/cli/cbcli/couchbase-cli-setting-notification.dita deleted file mode 100644 index 314f7c984c..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-notification.dita +++ /dev/null @@ -1,132 +0,0 @@ - - - setting-notification - - Modifies notification settings
    SYNTAX - - - couchbase-cli setting-notification [--cluster <url>] [--username <user>] - [--password <password>] [--enable-notification <int>]
    DESCRIPTION - - -

    Enabling notifications allow the Couchbase Administration Console to notify -users about newer versions of Couchbase.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---enable-notification <int> -
    -
    -

    - Specifies whether or not notifications should be enabled. To enable - notifications set this flag to "1". To disable notifications set this flag - to "0". -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To enable notifications on a Couchbase cluster run the following command\&.

    - 
    $ couchbase-cli setting-notification -c 192.168.1.5 -u Administrator \
- -p password --enable-notification 1
    -

    To disable notifications run the following command.

    - 
    $ couchbase-cli setting-notification -c 192.168.1.5 -u Administrator \
- -p password --enable-notification 0
    -
    DISCUSSION - - -

    The setting-notification subcommand either enables or disables notifications -about new Couchbase releases. This setting is a global parameter and applies to -all nodes in a cluster.

    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-notification.html b/content/cli/cbcli/couchbase-cli-setting-notification.html new file mode 100644 index 0000000000..bbb7e92277 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-notification.html @@ -0,0 +1,909 @@ + + + + + +couchbase-cli-setting-notification(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-notification [--cluster <url>] [--username <user>]
+          [--password <password>] [--enable-notification <int>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Enabling notifications allow the Couchbase Administration Console to notify +users about newer versions of Couchbase.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--enable-notification <int> +
    +
    +

    + Specifies whether or not notifications should be enabled. To enable + notifications set this flag to "1". To disable notifications set this flag + to "0". +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To enable notifications on a Couchbase cluster run the following command\&.

    +
    +
    +
    $ couchbase-cli setting-notification -c 192.168.1.5 -u Administrator \
+ -p password --enable-notification 1
    +
    +

    To disable notifications run the following command.

    +
    +
    +
    $ couchbase-cli setting-notification -c 192.168.1.5 -u Administrator \
+ -p password --enable-notification 0
    +
    +
    +
    +
    +

    DISCUSSION

    +
    +

    The setting-notification subcommand either enables or disables notifications +about new Couchbase releases. This setting is a global parameter and applies to +all nodes in a cluster.

    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli(1). +couchbase-cli-cluster-init(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-password-policy.dita b/content/cli/cbcli/couchbase-cli-setting-password-policy.dita deleted file mode 100644 index d2934c8772..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-password-policy.dita +++ /dev/null @@ -1,190 +0,0 @@ - - - setting-password-policy - - Manage the passowrd policy for new users
    SYNTAX - - - couchbase-cli setting-password-policy [--cluster <url>] [--username <user>] - [--password <password>] [--get] [--set] [--min-length <num>] [--uppercase] - [--lowercase] [--digits] [--special-chars]
    DESCRIPTION - - -

    Allows retrival of the current password policy and creating a new password -policy for new Couchbase users. This allows administrators to customize the -complexity of user passwords.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---get -
    -
    -

    - Retrieves the current password policy. -

    -
    -     - -
    ---set -
    -
    -

    - Creates a new password policy. Use the flags below to specify different - levels of password complexity required for new users and users who change - their existing passwords. -

    -
    -     - -
    ---min-length <num> -
    -
    -

    - Specifies the minimum password length for new passwords. -

    -
    -     - -
    ---uppercase -
    -
    -

    - Specifies that new passwords must contain at least one upper case letter. -

    -
    -     - -
    ---lowercase -
    -
    -

    - Specifies that new passwords must contain at least one lower case letter. -

    -
    -     - -
    ---digit -
    -
    -

    - Specifies that new passwords must contain at least one digit. -

    -
    -     - -
    ---special-char -
    -
    -

    - Specifies that new passwords must contain at least one special character. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To get the current password policy run the following command.

    - 
    $ couchbase-cli setting-password-policy -c 192.168.1.5 -u Administrator \
- -p password --get
    -

    To set a new password policy where the minimum password length is 10 character -and require that all passwords contain an upper case letter, lower case letter, -and a digit run the following command.

    - 
    $ couchbase-cli setting-password-policy -c 192.168.1.5 -u Administrator \
- -p password --set --min-length 10 --uppercase --lowercase --digits
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-password-policy.html b/content/cli/cbcli/couchbase-cli-setting-password-policy.html new file mode 100644 index 0000000000..4cf3a32142 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-password-policy.html @@ -0,0 +1,953 @@ + + + + + +couchbase-cli-setting-password-policy(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-password-policy [--cluster <url>] [--username <user>]
+          [--password <password>] [--get] [--set] [--min-length <num>] [--uppercase]
+          [--lowercase] [--digits] [--special-chars]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Allows retrival of the current password policy and creating a new password +policy for new Couchbase users. This allows administrators to customize the +complexity of user passwords.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--get +
    +
    +

    + Retrieves the current password policy. +

    +
    +
    +--set +
    +
    +

    + Creates a new password policy. Use the flags below to specify different + levels of password complexity required for new users and users who change + their existing passwords. +

    +
    +
    +--min-length <num> +
    +
    +

    + Specifies the minimum password length for new passwords. +

    +
    +
    +--uppercase +
    +
    +

    + Specifies that new passwords must contain at least one upper case letter. +

    +
    +
    +--lowercase +
    +
    +

    + Specifies that new passwords must contain at least one lower case letter. +

    +
    +
    +--digit +
    +
    +

    + Specifies that new passwords must contain at least one digit. +

    +
    +
    +--special-char +
    +
    +

    + Specifies that new passwords must contain at least one special character. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To get the current password policy run the following command.

    +
    +
    +
    $ couchbase-cli setting-password-policy -c 192.168.1.5 -u Administrator \
+ -p password --get
    +
    +

    To set a new password policy where the minimum password length is 10 character +and require that all passwords contain an upper case letter, lower case letter, +and a digit run the following command.

    +
    +
    +
    $ couchbase-cli setting-password-policy -c 192.168.1.5 -u Administrator \
+ -p password --set --min-length 10 --uppercase --lowercase --digits
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli(1). +couchbase-cli-user-manage(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-security.dita b/content/cli/cbcli/couchbase-cli-setting-security.dita deleted file mode 100644 index 1a21835645..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-security.dita +++ /dev/null @@ -1,121 +0,0 @@ - - - setting-security - - Manage security policies
    SYNTAX - - - couchbase-cli setting-password-policy [--cluster <url>] [--username <user>] - [--password <password>] [--disable-http-ui]
    DESCRIPTION - - -

    Allows enabling and disabling of the Couchbase Administration Console.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---disable-http-ui -
    -
    -

    - Disables the Couchbase Administration Console over HTTP. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To get disable the Couchbase Administration Console over HTTP run the following -command.

    - 
    $ couchbase-cli setting-security -c 192.168.1.5 -u Administrator \
- -p password --disable-http-ui
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-security.html b/content/cli/cbcli/couchbase-cli-setting-security.html new file mode 100644 index 0000000000..8c4bffa8be --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-security.html @@ -0,0 +1,895 @@ + + + + + +couchbase-cli-setting-security(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-password-policy [--cluster <url>] [--username <user>]
+          [--password <password>] [--disable-http-ui <0|1>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    Allows enabling and disabling of the Couchbase Administration Console.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--disable-http-ui <0|1> +
    +
    +

    + Specifies whether the Couchbase Administration UI can be accessible + over http. To disable access set this option to "1". To enable access + set this option to "0". By default, access to the UI over http is + enabled. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To disable the Couchbase Administration Console over HTTP run the following +command.

    +
    +
    +
    $ couchbase-cli setting-security -c 192.168.1.5 -u Administrator \
+ -p password --disable-http-ui 1
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli(1).

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-setting-xdcr.dita b/content/cli/cbcli/couchbase-cli-setting-xdcr.dita deleted file mode 100644 index 9c3539f12f..0000000000 --- a/content/cli/cbcli/couchbase-cli-setting-xdcr.dita +++ /dev/null @@ -1,233 +0,0 @@ - - - setting-xdcr - - Modifies cross datacenter replication (XDCR) settings
    SYNTAX - - - couchbase-cli setting-xdcr [--cluster <url>] [--username <user>] - [--password <password>] [--checkpoint-interval <seconds>] - [--worker-batch-size <num>] [--doc-batch-size <kilobytes>] - [--failure-restart-interval <seconds>] [--source-nozzle-per-node <num>] - [--target-nozzle-per-node <num>] [--bandwidth-usage-limit <num>] - [--stats-interval <milliseconds>] - [--optimistic-replication-threshold <bytes>] [--log-level <level>]
    DESCRIPTION - - -

    This command sets gloabl default settings for all XDCR replications. If you only -want to change the settings for a single XDCR replication see the - -couchbase-cli-xdcr-replicate1 - command.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---checkpoint-interval <seconds> -
    -
    -

    - The interval between checkpoints in seconds. The value of this option must - be between 60 and 14,400. -

    -
    -     - -
    ---worker-batch-size <num> -
    -
    -

    - The worker batch size. The value of this option must be between 500 and - 10,000. -

    -
    -     - -
    ---doc-batch-size <kilobytes> -
    -
    -

    - The document batch size in Kilobytes. The value of this option must be - between 10 and 100,000. -

    -
    -     - -
    ---failure-restart-interval <seconds> -
    -
    -

    - Interval for restarting failed XDCR connections in seconds. The value of - this option must be between 1 and 300. -

    -
    -     - -
    ---optimistic-replication-threshold <bytes> -
    -
    -

    - Document body size threshold in bytes used to trigger optimistic - replication. -

    -
    -     - -
    ---source-nozzle-per-node <num> -
    -
    -

    - The number of source nozzles to each node in the target cluster. The - value of this option must be between 1 and 10. -

    -
    -     - -
    ---target-nozzle-per-node <num> -
    -
    -

    - The number of outgoing nozzles to each node in the target cluster. The - value of this option must be between 1 and 10. -

    -
    -     - -
    ---bandwidth-usage-limit <num> -
    -
    -

    - The default bandwidth limit for XDCR replications in Megabytes per second. -

    -
    -     - -
    ---log-level <level> -
    -
    -

    - The XDCR log level. -

    -
    -     - -
    ---stats-interval <milliseconds> -
    -
    -

    - The interval for statistics updates in milliseconds. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    If we want to change the checkpoint interval to 500 seconds, the worker batch to -1000 documents, the document batch size to 10240KB, the failure restart interval -to 60 seconds. the optimistic replication threshold to 102400 bytes, the source -nozzels to 8, the target nozzles to 8, the log level to info, and the stats -interval to 100 milliseconds run the following command.

    - 
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
- -p password --checkpoint-interval 500 --worker-batch-size 1000 \
- --doc-batch-size 10240 --failure-restart-interval 60 \
- --optimistic-replication-threshold 102400 --source-nozzle-per-node 8 \
- --target-nozzle-per-node 8 --log-level info --stats-interval 100
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-setting-xdcr.html b/content/cli/cbcli/couchbase-cli-setting-xdcr.html new file mode 100644 index 0000000000..cb5050d3de --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-setting-xdcr.html @@ -0,0 +1,996 @@ + + + + + +couchbase-cli-setting-xdcr(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli setting-xdcr [--cluster <url>] [--username <user>]
+          [--password <password>] [--checkpoint-interval <seconds>]
+          [--worker-batch-size <num>] [--doc-batch-size <kilobytes>]
+          [--failure-restart-interval <seconds>] [--source-nozzle-per-node <num>]
+          [--target-nozzle-per-node <num>] [--bandwidth-usage-limit <num>]
+          [--enable-compression <num>] [--stats-interval <milliseconds>]
+          [--optimistic-replication-threshold <bytes>] [--log-level <level>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command sets global default settings for all XDCR replications. If you only +want to change the settings for a single XDCR replication see the +couchbase-cli-xdcr-replicate(1) command.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--checkpoint-interval <seconds> +
    +
    +

    + The interval between checkpoints in seconds. The value of this option must + be between 60 and 14,400. +

    +
    +
    +--worker-batch-size <num> +
    +
    +

    + The worker batch size. The value of this option must be between 500 and + 10,000. +

    +
    +
    +--doc-batch-size <kilobytes> +
    +
    +

    + The document batch size in Kilobytes. The value of this option must be + between 10 and 100,000. +

    +
    +
    +--failure-restart-interval <seconds> +
    +
    +

    + Interval for restarting failed XDCR connections in seconds. The value of + this option must be between 1 and 300. +

    +
    +
    +--optimistic-replication-threshold <bytes> +
    +
    +

    + Document body size threshold in bytes used to trigger optimistic + replication. +

    +
    +
    +--source-nozzle-per-node <num> +
    +
    +

    + The number of source nozzles to each node in the target cluster. The + value of this option must be between 1 and 10. +

    +
    +
    +--target-nozzle-per-node <num> +
    +
    +

    + The number of outgoing nozzles to each node in the target cluster. The + value of this option must be between 1 and 10. +

    +
    +
    +--bandwidth-usage-limit <num> +
    +
    +

    + The default bandwidth limit for XDCR replications in Megabytes per second. +

    +
    +
    +--enable-compression <num> +
    +
    +

    + Specifies whether or not XDCR compression is enabled. Set this option to + "1" to enable compression or "0" to disable compression. This feature is + only available in Couchbase Enterprise Edition and can only be used where + the target cluster supports compression. +

    +
    +
    +--log-level <level> +
    +
    +

    + The XDCR log level. +

    +
    +
    +--stats-interval <milliseconds> +
    +
    +

    + The interval for statistics updates in milliseconds. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    If we want to change the checkpoint interval to 500 seconds, the worker batch to +1000 documents, the document batch size to 10240KB, the failure restart interval +to 60 seconds. the optimistic replication threshold to 102400 bytes, the source +nozzles to 8, the target nozzles to 8, the log level to info, and the stats +interval to 100 milliseconds run the following command.

    +
    +
    +
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
+ -p password --checkpoint-interval 500 --worker-batch-size 1000 \
+ --doc-batch-size 10240 --failure-restart-interval 60 \
+ --optimistic-replication-threshold 102400 --source-nozzle-per-node 8 \
+ --target-nozzle-per-node 8 --log-level info --stats-interval 100
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-xdcr-replicate(1) +couchbase-cli-xdcr-setup(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-ssl-manage.dita b/content/cli/cbcli/couchbase-cli-ssl-manage.dita deleted file mode 100644 index 3789ddfe73..0000000000 --- a/content/cli/cbcli/couchbase-cli-ssl-manage.dita +++ /dev/null @@ -1,260 +0,0 @@ - - - ssl-manage - - Manage SSL certificates
    SYNTAX - - - couchbase-cli ssl-manage [--cluster <url>] [--username <user>] - [--password <password>] [--cluster-cert-info] [--node-cert-info] - [--regenerate-cert <file>] [--set-node-certificate] - [--upload-cluster-ca <file>] - [--set-client-auth-state <enable|disable|mandatory>] - [--set-client-auth-path <subject.cn|san.uri|san.dnsname|san.email>] - [--set-client-auth-prefix <prefix>] - [--set-client-auth-delimiter <delimiter>] - [--client-auth] [--extended]
    DESCRIPTION - - -

    This command allows users to manage their SSL and X.509 certificates.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---cluster-cert-info -
    -
    -

    - Retrieves the cluster certificate and prints it to stdout. -

    -
    -     - -
    ---node-cert-info -
    -
    -

    - Retrieves the node certificate and prints it to stdout. -

    -
    -     - -
    ---regenerate-cert <file> -
    -
    -

    - Regenerates the cluster certificate and saves it to the file specified in - the value fo this option. -

    -
    -     - -
    ---set-node-certificate -
    -
    -

    - Sets the node certificate. Before running this command you must copy the - node certificate into your installation. When this command is invoked the - cluster will read the node certificate from the node specified and set it - as the node certificate for that node. -

    -
    -     - -
    ---upload-cluster-ca <file> -
    -
    -

    - Uploads the certificate specified to the cluster. The uploaded certificate - will replace the cluster certificate in this cluster. -

    -
    -     - -
    ---set-client-auth-state <enable|disable|mandatory> -
    -
    -

    - Sets the client certificate auth to following values: -

    -
    • -

      -disabled : no client certification -

      -
    • -

      -enabled : if the client presents a valid client certificate then we try - and validate it. If the validation fails or no certificate is presented we - fall back to the existing authentication methods -

      -
    • -

      -mandatory : the client has to present valid SSL certificate in order to - access successfully authorise the connection. -

      -
    -
    -     -
    -

    --set-client-auth-path <path>: - The field which will be used to extract the username from the certificate - Currently only subject.cn, san.uri, san.email and san.dnsname are allowed

    -

    --set-client-auth-prefix <prefix> - Optional. Prefix to be ignored from the field value

    -

    --set-client-auth-delimiter <delimiter> - Optional. The delimiter can either be a string or a character, the parsing - of the username ends when the delimter value is found.

    -
    - -
    ---client-auth -
    -
    -

    - This options used to get the client cert auth value -

    -
    -     - -
    ---extended -
    -
    -

    - This option is used with the --cluster-cert-info and specifies that extended - cluster certificate information should be printed to stdout. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To get cluster certificate information run the following command.

    - 
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
- -p password --cluster-cert-info
    -

    Note that you can also get extended cluster certificat information by adding the ---extended flag as shown below.

    - 
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
- -p password --cluster-cert-info --extended
    -

    To get node certificate information run the following command. Note that the -node certificate will be from the node specified by the -c option.

    - 
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
- -p password --node-cert-info --extended
    -

    To automatically regenerate the cluster certificate and save the new certificate -to a file at /root/new_cluster.cert run the following command.

    - 
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
- -p password --regenerate-cert /root/new_cluster.cert
    -

    Note that the command above should never be run if you are using X.509 -certificates because using the --regenerate-cert command will generate an -unsigned certificate for the cluster to use.

    -

    To update a node certificate you will first need to copy the new certificate to -the certificate inbox folder on the node you wish to change the certificate. -Once you have done this you can run the command below to tell the server to -begin using the new certificate.

    - 
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
- -p password --set-node-certificate
    -

    To update the cluster certificate you can run the command below. Note that our -certificate is located at /root/new_cluster.cert in this example.

    - 
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
- -p password --upload-cluster-ca /root/new_cluster.cert
    -

    To set the client cert auth to mandatory, following command can be run.

    - 
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
- -p password --set-client-auth mandatory
    -

    To get the client cert auth value, following command should be run.

    - 
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
- -p password --client-auth
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-ssl-manage.html b/content/cli/cbcli/couchbase-cli-ssl-manage.html new file mode 100644 index 0000000000..b322c8ff13 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-ssl-manage.html @@ -0,0 +1,1077 @@ + + + + + +couchbase-cli-ssl-manage(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli ssl-manage [--cluster <url>] [--username <user>]
+          [--password <password>] [--cluster-cert-info] [--node-cert-info]
+          [--regenerate-cert <file>] [--set-node-certificate]
+          [--upload-cluster-ca <file>]
+          [--set-client-auth-state <enable|disable|mandatory>]
+          [--set-client-auth-path <subject.cn|san.uri|san.dnsname|san.email>]
+          [--set-client-auth-prefix <prefix>]
+          [--set-client-auth-delimiter <delimiter>]
+          [--client-auth] [--extended]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command allows users to manage their SSL and X.509 certificates.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--cluster-cert-info +
    +
    +

    + Retrieves the cluster certificate and prints it to stdout. +

    +
    +
    +--node-cert-info +
    +
    +

    + Retrieves the node certificate and prints it to stdout. +

    +
    +
    +--regenerate-cert <file> +
    +
    +

    + Regenerates the cluster certificate and saves it to the file specified in + the value fo this option. +

    +
    +
    +--set-node-certificate +
    +
    +

    + Sets the node certificate. Before running this command you must copy the + node certificate into your installation. When this command is invoked the + cluster will read the node certificate from the node specified and set it + as the node certificate for that node. +

    +
    +
    +--upload-cluster-ca <file> +
    +
    +

    + Uploads the certificate specified to the cluster. The uploaded certificate + will replace the cluster certificate in this cluster. +

    +
    +
    +--set-client-auth <path> +
    +
    +

    + Specifies a path to the client auth configuration file. This file should + contain the state of client auth and one or more prefixes. The state defines + whether or not client auth should used. The possible values for this field + are below: +

    +
      +
    • +

      +disabled: no client certification +

      +
      • +
    • +

      +enable: if the client presents a valid client certificate then we try + and validate it. If the validation fails or no certificate is presented we + fall back to the existing authentication methods +

      +
      • +
    • +

      +mandatory: the client has to present valid SSL certificate in order to + access successfully authorise the connection. +

      +
      +
      +
      The prefixes section should contain one or more prefixes and each prefix
+should contain a path, prefix, and delimiter. More information about these
+sub-fields is below:
      +
      +
      • +
    • +

      +path: The field which will be used to extract the username from the + certificate. Currently only subject.cn, san.uri, san.email and san.dnsname + are allowed +

      +
      • +
    • +

      +prefix: Optional. Prefix to be ignored from the field value +

      +
      • +
    • +

      +delimiter: Optional. The delimiter can either be a string or a character, + the parsing of the username ends when the delimter value is found. +

      +
      +
      +
      Below is an example of what a client auth configuraiton file might look
+like:
      +
      +
      +
      +
      {
+  "state": "enable",
+  "prefixes": [
+    {
+      "path": "subject.cn",
+      "prefix": "www.cb-",
+      "delimiter": "."
+    }
+  ]
+}
      +
      +
      • +
    +
    +
    +--client-auth +
    +
    +

    + This options used to get the client cert auth value +

    +
    +
    +--extended +
    +
    +

    + This option is used with the --cluster-cert-info and specifies that extended + cluster certificate information should be printed to stdout. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To get cluster certificate information run the following command.

    +
    +
    +
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
+ -p password --cluster-cert-info
    +
    +

    Note that you can also get extended cluster certificate information by adding the +--extended flag as shown below.

    +
    +
    +
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
+ -p password --cluster-cert-info --extended
    +
    +

    To get node certificate information run the following command. Note that the +node certificate will be from the node specified by the -c option.

    +
    +
    +
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
+ -p password --node-cert-info --extended
    +
    +

    To automatically regenerate the cluster certificate and save the new certificate +to a file at /root/new_cluster.cert run the following command.

    +
    +
    +
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
+ -p password --regenerate-cert /root/new_cluster.cert
    +
    +

    Note that the command above should never be run if you are using X.509 +certificates because using the --regenerate-cert command will generate an +unsigned certificate for the cluster to use.

    +

    To update a node certificate you will first need to copy the new certificate to +the certificate inbox folder on the node you wish to change the certificate. +Once you have done this you can run the command below to tell the server to +begin using the new certificate.

    +
    +
    +
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
+ -p password --set-node-certificate
    +
    +

    To update the cluster certificate you can run the command below. Note that our +certificate is located at /root/new_cluster.cert in this example.

    +
    +
    +
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
+ -p password --upload-cluster-ca /root/new_cluster.cert
    +
    +

    To set the client cert auth to mandatory, following command can be run.

    +
    +
    +
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
+ -p password --set-client-auth mandatory
    +
    +

    To get the client cert auth value, following command should be run.

    +
    +
    +
    $ couchbase-cli ssl-manage -c 192.168.1.5 -u Administrator \
+ -p password --client-auth
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-cluster-edit(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-user-manage.dita b/content/cli/cbcli/couchbase-cli-user-manage.dita deleted file mode 100644 index cdbb4a156a..0000000000 --- a/content/cli/cbcli/couchbase-cli-user-manage.dita +++ /dev/null @@ -1,468 +0,0 @@ - - - user-manage - - Manage RBAC users
    SYNTAX - - - couchbase-cli user-manage [--cluster <url>] [--username <user>] - [--password <password>] [--delete] [--list] [--my-roles] [--set] - [--rbac-username <username>] [--rbac-password <password>] - [--rbac-name <name>] [--roles <roles_list>] [--auth_domain <domain>]
    DESCRIPTION - - -

    This command allows administrators to assign and manage roles to different users -in their organization. Users can either be managed locally by Couchbase or -externally through the use of an external domain.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---delete -
    -
    -

    - Deletes an RBAC user profile from the cluster. You must have full - administrator privileges in order to delete a user profile. -

    -
    -     - -
    ---list -
    -
    -

    - Lists all RBAC user profiles in the cluster and show their roles. You must - have full administrator privileges in order to list all user profiles. -

    -
    -     - -
    ---my-roles -
    -
    -

    - Shows the current users RBAC user profile. -

    -
    -     - -
    ---set -
    -
    -

    - Creates or updates an RBAC user profile. You must have full administrator - privileges in order to create or update a user profile. -

    -
    -     - -
    ---rbac-username <username> -
    -
    -

    - Specifies the username of the RBAC user to modify. This option is used when - deleting, creating, or updating an RBAC user profile. -

    -
    -     - -
    ---rbac-password <password> -
    -
    -

    - Specifies the password to be used for an RBAC user profile. This option is - used only when creating or updating a local RBAC user profile. Couchbase - does not store password for external RBAC roles. -

    -
    -     - -
    ---rbac-name <name> -
    -
    -

    - Specifies the name to be used for an RBAC user profile. This option is used - when creating or updating an RBAC user profile and it is recommanded that - this option be set to the users full name. -

    -
    -     - -
    ---roles <roles_list> -
    -
    -

    - Specifies the roles to be given to an RBAC user profile. This option is used - when creating or updating an RBAC user profile and it is specified as a - comma separated list of roles. See the ROLES section for more details on the - available roles in Couchbase. -

    -
    -     - -
    ---auth-domain <domain> -
    -
    -

    - Specifies the auth_domain to used for an RBAC user profile. This option is - used when deleting, creating or updating an RBAC user profile and it if may - be set to either local or external. Loacl users are users that are - managed directly by the Couchbase cluster. External users are users - managed by an external source suchas LDAP. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -

    ROLES

    - -

    Cluster-Wide Roles:

    -
    -admin -
    -

    - Give the user permissions to manage all Couchbase configuration settings, - and read and write all data in the cluster. This user can make changes to - anything in the cluster. -

    -
    -bucket_admin[…] -
    -

    - Gives the user permissions to manage bucket settings. This role can be - assigned globally to all buckets or to a particular bucket. For XDCR - operations, the user can start/stop replication for the buckets they - administer, but they cannot set up the XDCR cluster references. To give - a user the ability to manage all bucket settings set their role to - bucket_admin[*]. To give the user permission to manage bucket settings on a - single bucket named default then specify the role as - bucket_admin[default]. If the user needs to be manage multiple buckets, for - example default and app, then set the role as bucket[default,app]. -

    -
    -cluster_admin -
    -

    - Gives the user permissions to read, write and manage all cluster-level - settings except security. -

    -
    -replication_admin -
    -

    - Allows the user to configure XDCR topology and manage XDCR replications. -

    -
    -ro_admin -
    -

    - Gives the user read-only access and cannot make any changes to the system. - This user has read-only access to cluster overview, design documents - (without the ability to create or query views), bucket summaries (without - the ability to create or view documents), XDCR cluster references, XDCR - replications, and cluster settings. -

    -
    -view_admin[…] -
    -

    - Gives the user privileges to define views and then run these views on data - to ensure that views are defined properly. This applies both to the - map-reduce and spatial views. To give a user the ability to manage views on - all buckets set their role to views_admin[*]. To give the user permission to - manage views on a single bucket named default then specify the role as - views_admin[default]. If the user needs to be manage views for multiple - buckets, for example default and app, then set the role as - views_admin[default,app]. -

    -
    -

    Data Service Roles:

    -
    -data_reader[…] -
    -

    - Gives the user permission to read data through Couchbases key-value APIs. - To give a user read-only access for all buckets set their role to - data_reader[*]. To give the user read-only access to data on a single - bucket named default then specify their role as data_reader[default]. - If the user needs read-only access to data for multiple buckets, for example - default and app, then set their role as data_reader[default,app]. -

    -
    -data_reader_writer[…] -
    -

    - Gives the user permission to read and write data through Couchbases - key-value APIs. The user cannot however modify the settings of a bucket. - To give a user read-write access for all buckets set their role to - data_reader_writer[*]. To give the user read-write access to data on a - single bucket named default then specify their role as - data_reader_writer[default]. If the user needs read-write access to data - for multiple buckets, for example default and app, then set their role - as data_reader_writer[default,app]. -

    -
    -data_dcp_reader[…] -
    -

    - Gives the user permission to create Couchbase DCP connections. To give a - user the ability to create DCP connections for all buckets set their role to - data_dcp_reader[*]. To give the user the ability to create DCP connections - on a single bucket named default then specify their role as - data_dcp_reader[default]. If the user needs to be able to create DCP - connections for multiple buckets, for example default and app, then set - their role as data_dcp_reader[default,app]. -

    -
    -data_backup[…] -
    -

    - Gives the user permission to backup and restore data in Couchbase. To give a - user the ability to backup and restore data for all buckets set their role - to data_backup[*]. To give the user the ability to backup and restore data - on a single bucket named default then specify their role as - data_backup[default]. If the user needs to be able to backup and restore - data for multiple buckets, for example default and app, then set their - role as data_backup[default,app]. -

    -
    -data_monitoring[…] -
    -

    - Gives the user permission to read monitoring data related to the data - service in Couchbase. To give a user the ability to monitor data for all - buckets set their role to data_monitoring[*]. To give the user the ability - to monitor data on a single bucket named default then specify their role - as data_monitoring[default]. If the user needs to be able to monitor data - for multiple buckets, for example default and app, then set their role - as data_monitoring[default,app]. -

    -
    -

    Full Text Service Roles:

    -
    -fts_admin[…] -
    -

    - Gives the user full administrator access for the Full Text Indexing service - for the specified buckets. To give a user full administrator access for FTS - on all buckets set their role to fts_admin[*]. To give the user full - administrator access for FTS on a single bucket named default then specify - their role as fts_admin[default]. If the user needs full administrator - access for FTS for multiple buckets, for example default and app, then - set their role as fts_admin[default,app]. -

    -
    -fts_searcher[…] -
    -

    - Allows the user to query full text indexes for the specified buckets. To - give a user the ability to query full text indexes on all buckets set their - role to fts_searcher[*]. To give the ability to query FTS indexes on a - single bucket named default then specify their role as - fts_searcher[default]. If the user needs to query FTS indexes on multiple - multiple buckets, for example default and app, then set their role as - fts_searcher[default,app]. -

    -
    -

    Query Service Roles:

    -
    -manage_index[…] -
    -

    - Allows the user to create and delete indexes on the specified buckets. To - give a user the ability to create and delete indexes on all buckets set - their role to manage_index[*]. To give the user permission to create and - delete indexes on a single bucket named default then specify their role - as manage_index[default]. If the user needs to be create and delete indexes - for multiple buckets, for example default and app, then set their role - as manage_index[default,app]. -

    -
    -query_delete[…] -
    -

    - Allows the user to execute DELETE query statements on the specified buckets. - To give a user the ability execute DELETE statements on all buckets set - their role to query_delete[*]. To give the user permission to execute - DELETE statements on a single bucket named default then specify their role - as query_delete[default]. If the user needs to be execute DELETE statements - for multiple buckets, for example default and app, then set their role - as query_delete[default,app]. -

    -
    -query_insert[…] -
    -

    - Allows the user to execute INSERT query statements on the specified buckets. - To give a user the ability execute INSERT statements on all buckets set - their role to query_insert[*]. To give the user permission to execute - INSERT statements on a single bucket named default then specify their role - as query_insert[default]. If the user needs to be execute INSERT statements - for multiple buckets, for example default and app, then set their role - as query_insert[default,app]. -

    -
    -query_select[…] -
    -

    - Allows the user to execute SELECT query statements on the specified buckets. - To give a user the ability execute SELECT statements on all buckets set - their role to query_select[*]. To give the user permission to execute - SELECT statements on a single bucket named default then specify their role - as query_select[default]. If the user needs to be execute SELECT statements - for multiple buckets, for example default and app, then set their role - as query_select[default,app]. -

    -
    -query_update[…] -
    -

    - Allows the user to execute UPDATE query statements on the specified buckets. - To give a user the ability execute UPDATE statements on all buckets set - their role to query_update[*]. To give the user permission to execute - UPDATE statements on a single bucket named default then specify their role - as query_update[default]. If the user needs to be execute UPDATE statements - for multiple buckets, for example default and app, then set their role - as query_update[default,app]. -

    -
    -system_catalog[…] -
    -

    - Allows the users to run queries against the system catalog on the specified - buckets. To give a user the ability to run queries against the system - catalog on all buckets set their role to system_catalog[*]. To give the user - permission to run queires against the system catalog on a single bucket - named default then specify their role as system_catalog[default]. If the - user needs to be run queries against the system catalog for multiple - buckets, for example default and app, then set their role as - system_catalog[default,app]. -

    -
    -     -
    EXAMPLES - - -

    To create an local RBAC user profile for a user named "John Doe" with username -jdoe and password cbpass with roles to manage the default bucket and all -XDCR replication run the following command

    - 
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
- -p password --set --rbac-username jdoe --rbac-password cbpass \
- --rbac-name "John Doe" --roles bucket_admin[default],replication_admin \
- --auth-domain local
    -

    If you have external user source setup in your cluster and you want to add a -user "John Doe" with username jdoe who should have the ability to manage only -views for all bucket run the following command

    - 
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
- -p password --set --rbac-username jdoe --rbac-name "John Doe" \
- --roles view_admin[*] --auth-domain external
    -

    To list the current RBAC user profiles run the following command.

    - 
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
- -p password --list
    -

    To delete an external user named jdoe run the following command.

    - 
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
- -p password --delete --rbac-username jdoe --auth-domain external
    -

    To delete a local user named jdoe run the following command.

    - 
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
- -p password --delete --rbac-username jdoe --auth-domain local
    -

    To see the user profile for a user with the username jdoe and password cbpass -run the following command.

    - 
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u jdoe -p cbpass \
- --my-roles
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-user-manage.html b/content/cli/cbcli/couchbase-cli-user-manage.html new file mode 100644 index 0000000000..2ef9ba137a --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-user-manage.html @@ -0,0 +1,1283 @@ + + + + + +couchbase-cli-user-manage(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli user-manage [--cluster <url>] [--username <user>]
+          [--password <password>] [--delete] [--list] [--my-roles] [--set]
+          [--rbac-username <username>] [--rbac-password <password>]
+          [--rbac-name <name>] [--roles <roles_list>] [--auth_domain <domain>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command allows administrators to assign and manage roles to different users +in their organization. Users can either be managed locally by Couchbase or +externally through the use of an external domain.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--delete +
    +
    +

    + Deletes an RBAC user profile from the cluster. You must have full + administrator privileges in order to delete a user profile. +

    +
    +
    +--list +
    +
    +

    + Lists all RBAC user profiles in the cluster and show their roles. You must + have full administrator privileges in order to list all user profiles. +

    +
    +
    +--my-roles +
    +
    +

    + Shows the current users RBAC user profile. +

    +
    +
    +--set +
    +
    +

    + Creates or updates an RBAC user profile. You must have full administrator + privileges in order to create or update a user profile. +

    +
    +
    +--rbac-username <username> +
    +
    +

    + Specifies the username of the RBAC user to modify. This option is used when + deleting, creating, or updating an RBAC user profile. +

    +
    +
    +--rbac-password <password> +
    +
    +

    + Specifies the password to be used for an RBAC user profile. This option is + used only when creating or updating a local RBAC user profile. Couchbase + does not store password for external RBAC roles. +

    +
    +
    +--rbac-name <name> +
    +
    +

    + Specifies the name to be used for an RBAC user profile. This option is used + when creating or updating an RBAC user profile and it is recommanded that + this option be set to the users full name. +

    +
    +
    +--roles <roles_list> +
    +
    +

    + Specifies the roles to be given to an RBAC user profile. This option is used + when creating or updating an RBAC user profile and it is specified as a + comma separated list of roles. See the ROLES section for more details on the + available roles in Couchbase. +

    +
    +
    +--auth-domain <domain> +
    +
    +

    + Specifies the auth_domain to used for an RBAC user profile. This option is + used when deleting, creating or updating an RBAC user profile and it if may + be set to either local or external. Loacl users are users that are + managed directly by the Couchbase cluster. External users are users + managed by an external source suchas LDAP. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +

    ROLES

    +
    +
    +

    Cluster-Wide Roles:

    +
    +
    +admin +
    +
    +

    + Give the user permissions to manage all Couchbase configuration settings, + and read and write all data in the cluster. This user can make changes to + anything in the cluster. +

    +
    +
    +bucket_admin[…] +
    +
    +

    + Gives the user permissions to manage bucket settings. This role can be + assigned globally to all buckets or to a particular bucket. For XDCR + operations, the user can start/stop replication for the buckets they + administer, but they cannot set up the XDCR cluster references. To give + a user the ability to manage all bucket settings set their role to + bucket_admin[*]. To give the user permission to manage bucket settings on a + single bucket named default then specify the role as + bucket_admin[default]. If the user needs to be manage multiple buckets, for + example default and app, then set the role as bucket[default,app]. +

    +
    +
    +cluster_admin +
    +
    +

    + Gives the user permissions to read, write and manage all cluster-level + settings except security. +

    +
    +
    +replication_admin +
    +
    +

    + Allows the user to configure XDCR topology and manage XDCR replications. +

    +
    +
    +ro_admin +
    +
    +

    + Gives the user read-only access and cannot make any changes to the system. + This user has read-only access to cluster overview, design documents + (without the ability to create or query views), bucket summaries (without + the ability to create or view documents), XDCR cluster references, XDCR + replications, and cluster settings. +

    +
    +
    +view_admin[…] +
    +
    +

    + Gives the user privileges to define views and then run these views on data + to ensure that views are defined properly. This applies both to the + map-reduce and spatial views. To give a user the ability to manage views on + all buckets set their role to views_admin[*]. To give the user permission to + manage views on a single bucket named default then specify the role as + views_admin[default]. If the user needs to be manage views for multiple + buckets, for example default and app, then set the role as + views_admin[default,app]. +

    +
    +
    +

    Data Service Roles:

    +
    +
    +data_reader[…] +
    +
    +

    + Gives the user permission to read data through Couchbases key-value APIs. + To give a user read-only access for all buckets set their role to + data_reader[*]. To give the user read-only access to data on a single + bucket named default then specify their role as data_reader[default]. + If the user needs read-only access to data for multiple buckets, for example + default and app, then set their role as data_reader[default,app]. +

    +
    +
    +data_writer[…] +
    +
    +

    + Gives the user permission to read and write data through Couchbases + key-value APIs. The user cannot however modify the settings of a bucket. + To give a user read-write access for all buckets set their role to + data_writer[*]. To give the user read-write access to data on a single + bucket named default then specify their role as data_writer[default]. + If the user needs read-write access to data for multiple buckets, for + example default and app, then set their role as + data_writer[default,app]. +

    +
    +
    +data_dcp_reader[…] +
    +
    +

    + Gives the user permission to create Couchbase DCP connections. To give a + user the ability to create DCP connections for all buckets set their role to + data_dcp_reader[*]. To give the user the ability to create DCP connections + on a single bucket named default then specify their role as + data_dcp_reader[default]. If the user needs to be able to create DCP + connections for multiple buckets, for example default and app, then set + their role as data_dcp_reader[default,app]. +

    +
    +
    +data_backup[…] +
    +
    +

    + Gives the user permission to backup and restore data in Couchbase. To give a + user the ability to backup and restore data for all buckets set their role + to data_backup[*]. To give the user the ability to backup and restore data + on a single bucket named default then specify their role as + data_backup[default]. If the user needs to be able to backup and restore + data for multiple buckets, for example default and app, then set their + role as data_backup[default,app]. +

    +
    +
    +data_monitoring[…] +
    +
    +

    + Gives the user permission to read monitoring data related to the data + service in Couchbase. To give a user the ability to monitor data for all + buckets set their role to data_monitoring[*]. To give the user the ability + to monitor data on a single bucket named default then specify their role + as data_monitoring[default]. If the user needs to be able to monitor data + for multiple buckets, for example default and app, then set their role + as data_monitoring[default,app]. +

    +
    +
    +

    Full Text Service Roles:

    +
    +
    +fts_admin[…] +
    +
    +

    + Gives the user full administrator access for the Full Text Indexing service + for the specified buckets. To give a user full administrator access for FTS + on all buckets set their role to fts_admin[*]. To give the user full + administrator access for FTS on a single bucket named default then specify + their role as fts_admin[default]. If the user needs full administrator + access for FTS for multiple buckets, for example default and app, then + set their role as fts_admin[default,app]. +

    +
    +
    +fts_searcher[…] +
    +
    +

    + Allows the user to query full text indexes for the specified buckets. To + give a user the ability to query full text indexes on all buckets set their + role to fts_searcher[*]. To give the ability to query FTS indexes on a + single bucket named default then specify their role as + fts_searcher[default]. If the user needs to query FTS indexes on multiple + multiple buckets, for example default and app, then set their role as + fts_searcher[default,app]. +

    +
    +
    +

    Query Service Roles:

    +
    +
    +manage_index[…] +
    +
    +

    + Allows the user to create and delete indexes on the specified buckets. To + give a user the ability to create and delete indexes on all buckets set + their role to manage_index[*]. To give the user permission to create and + delete indexes on a single bucket named default then specify their role + as manage_index[default]. If the user needs to be create and delete indexes + for multiple buckets, for example default and app, then set their role + as manage_index[default,app]. +

    +
    +
    +query_delete[…] +
    +
    +

    + Allows the user to execute DELETE query statements on the specified buckets. + To give a user the ability execute DELETE statements on all buckets set + their role to query_delete[*]. To give the user permission to execute + DELETE statements on a single bucket named default then specify their role + as query_delete[default]. If the user needs to be execute DELETE statements + for multiple buckets, for example default and app, then set their role + as query_delete[default,app]. +

    +
    +
    +query_insert[…] +
    +
    +

    + Allows the user to execute INSERT query statements on the specified buckets. + To give a user the ability execute INSERT statements on all buckets set + their role to query_insert[*]. To give the user permission to execute + INSERT statements on a single bucket named default then specify their role + as query_insert[default]. If the user needs to be execute INSERT statements + for multiple buckets, for example default and app, then set their role + as query_insert[default,app]. +

    +
    +
    +query_select[…] +
    +
    +

    + Allows the user to execute SELECT query statements on the specified buckets. + To give a user the ability execute SELECT statements on all buckets set + their role to query_select[*]. To give the user permission to execute + SELECT statements on a single bucket named default then specify their role + as query_select[default]. If the user needs to be execute SELECT statements + for multiple buckets, for example default and app, then set their role + as query_select[default,app]. +

    +
    +
    +query_update[…] +
    +
    +

    + Allows the user to execute UPDATE query statements on the specified buckets. + To give a user the ability execute UPDATE statements on all buckets set + their role to query_update[*]. To give the user permission to execute + UPDATE statements on a single bucket named default then specify their role + as query_update[default]. If the user needs to be execute UPDATE statements + for multiple buckets, for example default and app, then set their role + as query_update[default,app]. +

    +
    +
    +system_catalog[…] +
    +
    +

    + Allows the users to run queries against the system catalog on the specified + buckets. To give a user the ability to run queries against the system + catalog on all buckets set their role to system_catalog[*]. To give the user + permission to run queries against the system catalog on a single bucket + named default then specify their role as system_catalog[default]. If the + user needs to be run queries against the system catalog for multiple + buckets, for example default and app, then set their role as + system_catalog[default,app]. +

    +
    +
    +
    +
    +
    +
    +
    +
    +

    EXAMPLES

    +
    +

    To create an local RBAC user profile for a user named "John Doe" with username +jdoe and password cbpass with roles to manage the default bucket and all +XDCR replication run the following command

    +
    +
    +
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
+ -p password --set --rbac-username jdoe --rbac-password cbpass \
+ --rbac-name "John Doe" --roles bucket_admin[default],replication_admin \
+ --auth-domain local
    +
    +

    If you have external user source setup in your cluster and you want to add a +user "John Doe" with username jdoe who should have the ability to manage only +views for all bucket run the following command

    +
    +
    +
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
+ -p password --set --rbac-username jdoe --rbac-name "John Doe" \
+ --roles view_admin[*] --auth-domain external
    +
    +

    To list the current RBAC user profiles run the following command.

    +
    +
    +
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
+ -p password --list
    +
    +

    To delete an external user named jdoe run the following command.

    +
    +
    +
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
+ -p password --delete --rbac-username jdoe --auth-domain external
    +
    +

    To delete a local user named jdoe run the following command.

    +
    +
    +
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u Administrator \
+ -p password --delete --rbac-username jdoe --auth-domain local
    +
    +

    To see the user profile for a user with the username jdoe and password cbpass +run the following command.

    +
    +
    +
    $ couchbase-cli user-manage -c 127.0.0.1:8091 -u jdoe -p cbpass \
+ --my-roles
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-ldap(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-xdcr-replicate.dita b/content/cli/cbcli/couchbase-cli-xdcr-replicate.dita deleted file mode 100644 index 6050c318b7..0000000000 --- a/content/cli/cbcli/couchbase-cli-xdcr-replicate.dita +++ /dev/null @@ -1,389 +0,0 @@ - - - xdcr-replicate - - Creates a replication between two datacenters
    SYNTAX - - - couchbase-cli xdcr-replicate [--cluster <url>] [--username <user>] - [--password <password>] [--create] [--delete] [--pause] [--list] - [--resume] [--settings] [--xdcr-from-bucket <bucket>] - [--xdcr-to-bucket <bucket>] [--xdcr-cluster-name <name>] - [--xdcr-replication-mode <mode>] [--filter-expression <regex>] - [--xdcr-replicator <id>] [--checkpoint-interval <seconds>] - [--worker-batch-size <num>] [--doc-batch-size <kilobytes>] - [--failure-restart-interval <seconds>] [--source-nozzle-per-node <num>] - [--target-nozzle-per-node <num>] [--bandwidth-usage-limit <num>] - [--stats-interval <milliseconds>] - [--optimistic-replication-threshold <bytes>] [--log-level <level>]
    DESCRIPTION - - -

    This command is used to manage XDCR replications.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---create -
    -
    -

    - Creates a new XDCR replication. -

    -
    -     - -
    ---delete -
    -
    -

    - Deletes an XDCR replication. -

    -
    -     - -
    ---pause -
    -
    -

    - Pauses an XDCR replication. -

    -
    -     - -
    ---list -
    -
    -

    - Lists all XDCR replications. -

    -
    -     - -
    ---resume -
    -
    -

    - Resumes an XDCR replication. -

    -
    -     - -
    ---settings -
    -
    -

    - Sets advanced settings for an XDCR replication. -

    -
    -     - -
    ---xdcr-from-bucket <bucket> -
    -
    -

    - The name bucket to replicate data from. -

    -
    -     - -
    ---xdcr-to-bucket <bucket> -
    -
    -

    - The name bucket to replicate data to. -

    -
    -     - -
    ---xdcr-cluster-name <name> -
    -
    -

    - The name of the cluster reference to replicate to. -

    -
    -     - -
    ---xdcr-replication-mode <mode> -
    -
    -

    - The XDCR replication protocol to use. This can be set to either "capi" or - "xmem". -

    -
    -     - -
    ---filter-expression <regex> -
    -
    -

    - A regular expression used to filter the replication stream. -

    -
    -     - -
    ---xdcr-replicator <id> -
    -
    -

    - The XDCR Replication ID. To get a list of replicator ID’s use the --list - flag. -

    -
    -     - -
    ---checkpoint-interval <seconds> -
    -
    -

    - The interval between checkpoints in seconds. The value of this option must - be between 60 and 14,400. -

    -
    -     - -
    ---worker-batch-size <num> -
    -
    -

    - The worker batch size. The value of this option must be between 500 and - 10,000. -

    -
    -     - -
    ---doc-batch-size <kilobytes> -
    -
    -

    - The document batch size in Kilobytes. The value of this option must be - between 10 and 100,000. -

    -
    -     - -
    ---failure-restart-interval <seconds> -
    -
    -

    - Interval for restarting failed XDCR connections in seconds. The value of - this option must be between 1 and 300. -

    -
    -     - -
    ---optimistic-replication-threshold <bytes> -
    -
    -

    - Document body size threshold in bytes used to trigger optimistic - replication. -

    -
    -     - -
    ---source-nozzle-per-node <num> -
    -
    -

    - The number of source nozzles to each node in the target cluster. The - value of this option must be between 1 and 10. -

    -
    -     - -
    ---target-nozzle-per-node <num> -
    -
    -

    - The number of outgoing nozzles to each node in the target cluster. The - value of this option must be between 1 and 10. -

    -
    -     - -
    ---bandwidth-usage-limit <num> -
    -
    -

    - The bandwidth limit for XDCR replications in Megabytes per second for this - replication. -

    -
    -     - -
    ---log-level <level> -
    -
    -

    - The XDCR log level. -

    -
    -     - -
    ---stats-interval <milliseconds> -
    -
    -

    - The interval for statistics updates in milliseconds. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To create a new XDCR replication from the "default" bucket to the "apps" bucket -on a remote cluster called "east" using the xmem replication protocol. You can -run the following command below. Note that if you have not setup a remote -cluster reference then you need to do this first by running the - -couchbase-cli-xdcr-setup1 -.

    - 
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
- -p password --create --xdcr-cluster-name east --xdcr-from-bucket apps \
- --xdcr-to-bucket apps --xdcr-replication-mode xmem
    -

    To list all of the current XDCR replication you can run the following command.

    - 
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
- -p password --list
    -

    To delete an XDCR replication you first need to use the --list flag to get the -replicator id. Once you get the replicator id (in this case we will assume it is -f4eb540d74c43fd3ac6d4b7910c8c92f/default/default) you can run the command below -to delete the replication.

    - 
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
- -p password --delete \
- --xdcr-replicator=f4eb540d74c43fd3ac6d4b7910c8c92f/default/default
    -

    To pause an XDCR replication you first need to use the --list flag to get the -replicator id. Once you get the replicator id (in this case we will assume it is -f4eb540d74c43fd3ac6d4b7910c8c92f/default/default) you can run the command below -to pause the replication.

    - 
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
- -p password --pause \
- --xdcr-replicator=f4eb540d74c43fd3ac6d4b7910c8c92f/default/default
    -

    To resume an XDCR replication you first need to use the --list flag to get the -replicator id. Once you get the replicator id (in this case we will assume it is -f4eb540d74c43fd3ac6d4b7910c8c92f/default/default) you can run the command below -to resume the replication.

    - 
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
- -p password --resume \
- --xdcr-replicator=f4eb540d74c43fd3ac6d4b7910c8c92f/default/default
    -

    To modify the settings of an XDCR replication you first need to use the --list -flag to get the replicator id. Once you get the replicator id (in this case we -will assume it is f4eb540d74c43fd3ac6d4b7910c8c92f/default/default) you can run -the command if for example you wanted to change the document batch size to 2048 -and failure restart interval to 60 seconds.

    - 
       $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
-    -p password --settings --failure-restart-interval=60 \
-    --xdcr-replicator=f4eb540d74c43fd3ac6d4b7910c8c92f/default/default \
---doc-batch-size=2048
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-xdcr-replicate.html b/content/cli/cbcli/couchbase-cli-xdcr-replicate.html new file mode 100644 index 0000000000..d6dd962f68 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-xdcr-replicate.html @@ -0,0 +1,1143 @@ + + + + + +couchbase-cli-xdcr-replicate(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli xdcr-replicate [--cluster <url>] [--username <user>]
+          [--password <password>] [--create] [--delete] [--pause] [--list]
+          [--resume] [--settings] [--xdcr-from-bucket <bucket>]
+          [--xdcr-to-bucket <bucket>] [--xdcr-cluster-name <name>]
+          [--xdcr-replication-mode <mode>] [--filter-expression <regex>]
+          [--xdcr-replicator <id>] [--checkpoint-interval <seconds>]
+          [--worker-batch-size <num>] [--doc-batch-size <kilobytes>]
+          [--failure-restart-interval <seconds>] [--source-nozzle-per-node <num>]
+          [--target-nozzle-per-node <num>] [--bandwidth-usage-limit <num>]
+          [--enable-compression <num>] [--stats-interval <milliseconds>]
+          [--optimistic-replication-threshold <bytes>] [--log-level <level>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command is used to manage XDCR replications.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--create +
    +
    +

    + Creates a new XDCR replication. +

    +
    +
    +--delete +
    +
    +

    + Deletes an XDCR replication. +

    +
    +
    +--pause +
    +
    +

    + Pauses an XDCR replication. +

    +
    +
    +--list +
    +
    +

    + Lists all XDCR replications. +

    +
    +
    +--resume +
    +
    +

    + Resumes an XDCR replication. +

    +
    +
    +--settings +
    +
    +

    + Sets advanced settings for an XDCR replication. +

    +
    +
    +--xdcr-from-bucket <bucket> +
    +
    +

    + The name bucket to replicate data from. +

    +
    +
    +--xdcr-to-bucket <bucket> +
    +
    +

    + The name bucket to replicate data to. +

    +
    +
    +--xdcr-cluster-name <name> +
    +
    +

    + The name of the cluster reference to replicate to. +

    +
    +
    +--xdcr-replication-mode <mode> +
    +
    +

    + The XDCR replication protocol to use. This can be set to either "capi" or + "xmem". +

    +
    +
    +--filter-expression <regex> +
    +
    +

    + A regular expression used to filter the replication stream. +

    +
    +
    +--xdcr-replicator <id> +
    +
    +

    + The XDCR Replication ID. To get a list of replicator ID’s use the --list + flag. +

    +
    +
    +--checkpoint-interval <seconds> +
    +
    +

    + The interval between checkpoints in seconds. The value of this option must + be between 60 and 14,400. +

    +
    +
    +--worker-batch-size <num> +
    +
    +

    + The worker batch size. The value of this option must be between 500 and + 10,000. +

    +
    +
    +--doc-batch-size <kilobytes> +
    +
    +

    + The document batch size in Kilobytes. The value of this option must be + between 10 and 100,000. +

    +
    +
    +--failure-restart-interval <seconds> +
    +
    +

    + Interval for restarting failed XDCR connections in seconds. The value of + this option must be between 1 and 300. +

    +
    +
    +--optimistic-replication-threshold <bytes> +
    +
    +

    + Document body size threshold in bytes used to trigger optimistic + replication. +

    +
    +
    +--source-nozzle-per-node <num> +
    +
    +

    + The number of source nozzles to each node in the target cluster. The + value of this option must be between 1 and 10. +

    +
    +
    +--target-nozzle-per-node <num> +
    +
    +

    + The number of outgoing nozzles to each node in the target cluster. The + value of this option must be between 1 and 10. +

    +
    +
    +--bandwidth-usage-limit <num> +
    +
    +

    + The bandwidth limit for XDCR replications in Megabytes per second for this + replication. +

    +
    +
    +--enable-compression <num> +
    +
    +

    + Specifies whether or not XDCR compression is enabled. Set this option to + "1" to enable compression or "0" to disable compression. This feature is + only available in Couchbase Enterprise Edition and can only be used where + the target cluster supports compression. +

    +
    +
    +--log-level <level> +
    +
    +

    + The XDCR log level. +

    +
    +
    +--stats-interval <milliseconds> +
    +
    +

    + The interval for statistics updates in milliseconds. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To create a new XDCR replication from the "default" bucket to the "apps" bucket +on a remote cluster called "east" using the xmem replication protocol. You can +run the following command below. Note that if you have not setup a remote +cluster reference then you need to do this first by running the +couchbase-cli-xdcr-setup(1).

    +
    +
    +
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
+ -p password --create --xdcr-cluster-name east --xdcr-from-bucket apps \
+ --xdcr-to-bucket apps --xdcr-replication-mode xmem
    +
    +

    To list all of the current XDCR replication you can run the following command.

    +
    +
    +
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
+ -p password --list
    +
    +

    To delete an XDCR replication you first need to use the --list flag to get the +replicator id. Once you get the replicator id (in this case we will assume it is +f4eb540d74c43fd3ac6d4b7910c8c92f/default/default) you can run the command below +to delete the replication.

    +
    +
    +
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
+ -p password --delete \
+ --xdcr-replicator=f4eb540d74c43fd3ac6d4b7910c8c92f/default/default
    +
    +

    To pause an XDCR replication you first need to use the --list flag to get the +replicator id. Once you get the replicator id (in this case we will assume it is +f4eb540d74c43fd3ac6d4b7910c8c92f/default/default) you can run the command below +to pause the replication.

    +
    +
    +
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
+ -p password --pause \
+ --xdcr-replicator=f4eb540d74c43fd3ac6d4b7910c8c92f/default/default
    +
    +

    To resume an XDCR replication you first need to use the --list flag to get the +replicator id. Once you get the replicator id (in this case we will assume it is +f4eb540d74c43fd3ac6d4b7910c8c92f/default/default) you can run the command below +to resume the replication.

    +
    +
    +
    $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
+ -p password --resume \
+ --xdcr-replicator=f4eb540d74c43fd3ac6d4b7910c8c92f/default/default
    +
    +

    To modify the settings of an XDCR replication you first need to use the --list +flag to get the replicator id. Once you get the replicator id (in this case we +will assume it is f4eb540d74c43fd3ac6d4b7910c8c92f/default/default) you can run +the command if for example you wanted to change the document batch size to 2048 +and failure restart interval to 60 seconds.

    +
    +
    +
       $ couchbase-cli xdcr-replicate -c 192.168.1.5 -u Administrator \
+    -p password --settings --failure-restart-interval=60 \
+    --xdcr-replicator=f4eb540d74c43fd3ac6d4b7910c8c92f/default/default \
+--doc-batch-size=2048
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-xdcr(1) +couchbase-cli-xdcr-setup(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli-xdcr-setup.dita b/content/cli/cbcli/couchbase-cli-xdcr-setup.dita deleted file mode 100644 index 3ded3c42ac..0000000000 --- a/content/cli/cbcli/couchbase-cli-xdcr-setup.dita +++ /dev/null @@ -1,256 +0,0 @@ - - - xdcr-setup - - Manage references to remote clusters
    SYNTAX - - - couchbase-cli xdcr-setup [--cluster <url>] [--username <user>] - [--password <password>] [--create] [--delete] [--edit] [--list] - [--xdcr-cluster-name <name>] [--xdcr-hostname <hostname>] - [--xdcr-username <username>] [--xdcr-password <password>] - [--xdcr-demand-encryption <num>] [--xdcr-encryption-type <type>] - [--xdcr-certificate <file>]
    DESCRIPTION - - -

    This command is used to manage the remote clusters that are available to be -replicated to.

    -
    OPTIONS - - -
    - -
    --c -
    -
    ---cluster -
    -
    -

    - Specfies the hostname of a node in the cluster. See the HOST FORMATS - section for more information on specifying a hostname. -

    -
    -     - -
    --u -
    -
    ---user <username> -
    -
    -

    - Specifies the username of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. -

    -
    -     - -
    --p -
    -
    ---password <password> -
    -
    -

    - Specifies the password of the user executing the command. If you do not have - a user account with permision to execute the command then it will fail with - an unauthorized error. If this argument is specified, but no password is - given then the command will prompt the user for a password through - non-echoed stdin. You may also specify your password by using the - environment variable CB_REST_PASSWORD. -

    -
    -     - -
    ---create -
    -
    -

    - Creates an new XDCR remote reference. -

    -
    -     - -
    ---delete -
    -
    -

    - Deletes an XDCR remote reference. -

    -
    -     - -
    ---edit -
    -
    -

    - Edits an XDCR remote reference. -

    -
    -     - -
    ---list -
    -
    -

    - List all XDCR remote references. -

    -
    -     - -
    ---xdcr-cluster-name <name> -
    -
    -

    - The name for the remote cluster reference. -

    -
    -     - -
    ---xdcr-hostname <hostname> -
    -
    -

    - The hostname of the remote cluster reference. -

    -
    -     - -
    ---xdcr-username <username> -
    -
    -

    - The username of the remote cluster reference. -

    -
    -     - -
    ---xdcr-password <password> -
    -
    -

    - The password of the remote cluster reference. -

    -
    -     - -
    ---xdcr-demand-encryption <num> -
    -
    -

    - Enable SSL when replicating with this cluster. To enable SSL set the value - of this otion to 1. To disable SSL then set this option to 0. BY default the - value of this option is set to 0. -

    -
    -     - -
    ---xdcr-encryption-type <type> -
    -
    -

    - Specifes the type of encryption to use if encryption is enabled. This flag - may either be set to "half" encryption or "full encryption". Half encryption - means that passwords are encrypted, but data is not. This results in faster - data transfer, but less security. Full encryption means that all data and - passwords are encrypted which increases security, but reduces overall data - transfer speed. This flag does not have any affect if encryption is not - enabled. If encryption is enabled and this flag is set then the value - defaults to full encryption. -

    -
    -     - -
    ---xdcr-certificate <file> -
    -
    -

    - The path to the certificate used for encryption. -

    -
    -     -
    -
    HOST FORMATS - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    -
    EXAMPLES - - -

    To create a new remote reference to a Couchbase cluster named "east" run the -following command.

    - 
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
- -p password --create --xdcr-cluster-name east --xdcr-hostname 192.168.1.6
- --xdcr-username Administrator --xdcr-password password
    -

    If the new remote reference should always be encrypted then make sure to enable -encryption.

    - 
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
- -p password --create --xdcr-cluster-name east --xdcr-hostname 192.168.1.6 \
- --xdcr-username Administrator --xdcr-password password \
- --xdcr-demand-encryption 1 --xdcr-certificate /root/cluster.cert
    -

    To list all current XDCR cluster references run the following command.

    - 
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
- -p password --list
    -

    If you need to edit a cluster references named "east" and change the password -run the following command.

    - 
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
- -p password --edit --xdcr-cluster-name east --xdcr-hostname 192.168.1.6
- --xdcr-username Administrator --xdcr-password new_password
    -

    Note in the above example that you need to specify all of the current unchanging -configuration parameters also to avoid them being reset to defaults.

    -

    If you no longer need an XDCR remote reference then you can delete it. We should -this below using the "east" remote reference as an exmaple.

    - 
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
- -p password --delete --xdcr-cluster-name east
    -
    ENVIRONMENT AND CONFIGURATION VARIABLES - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    -
    -     -     \ No newline at end of file diff --git a/content/cli/cbcli/couchbase-cli-xdcr-setup.html b/content/cli/cbcli/couchbase-cli-xdcr-setup.html new file mode 100644 index 0000000000..fe75fa9e43 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli-xdcr-setup.html @@ -0,0 +1,1008 @@ + + + + + +couchbase-cli-xdcr-setup(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli xdcr-setup [--cluster <url>] [--username <user>]
+          [--password <password>] [--create] [--delete] [--edit] [--list]
+          [--xdcr-cluster-name <name>] [--xdcr-hostname <hostname>]
+          [--xdcr-username <username>] [--xdcr-password <password>]
+          [--xdcr-secure-connection <type>] [--xdcr-certificate <file>]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    This command is used to manage the remote clusters that are available to be +replicated to.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +-c +
    +
    +--cluster +
    +
    +

    + Specifies the hostname of a node in the cluster. See the HOST FORMATS + section for more information on specifying a hostname. +

    +
    +
    +-u +
    +
    +--user <username> +
    +
    +

    + Specifies the username of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. +

    +
    +
    +-p +
    +
    +--password <password> +
    +
    +

    + Specifies the password of the user executing the command. If you do not have + a user account with permission to execute the command then it will fail with + an unauthorized error. If this argument is specified, but no password is + given then the command will prompt the user for a password through + non-echoed stdin. You may also specify your password by using the + environment variable CB_REST_PASSWORD. +

    +
    +
    +--create +
    +
    +

    + Creates an new XDCR remote reference. +

    +
    +
    +--delete +
    +
    +

    + Deletes an XDCR remote reference. +

    +
    +
    +--edit +
    +
    +

    + Edits an XDCR remote reference. +

    +
    +
    +--list +
    +
    +

    + List all XDCR remote references. +

    +
    +
    +--xdcr-cluster-name <name> +
    +
    +

    + The name for the remote cluster reference. +

    +
    +
    +--xdcr-hostname <hostname> +
    +
    +

    + The hostname of the remote cluster reference. +

    +
    +
    +--xdcr-username <username> +
    +
    +

    + The username of the remote cluster reference. +

    +
    +
    +--xdcr-password <password> +
    +
    +

    + The password of the remote cluster reference. +

    +
    +
    +--xdcr-secure-connection <type> +
    +
    +

    + Specifies the type of encryption to use. This flag may either be set to + either "half", "full", or "none". Half encryption means that passwords are + encrypted, but data is not. This results in faster data transfer, but less + security. Full encryption means that all data and passwords are encrypted + which increases security, but reduces overall data transfer speed. If no + encryption is needed then "none" can be specified. This flag defaults to + "none" if it is not specified. +

    +
    +
    +--xdcr-certificate <file> +
    +
    +

    + The path to the certificate used for encryption. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    EXAMPLES

    +
    +

    To create a new remote reference to a Couchbase cluster named "east" run the +following command.

    +
    +
    +
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
+ -p password --create --xdcr-cluster-name east --xdcr-hostname 192.168.1.6
+ --xdcr-username Administrator --xdcr-password password
    +
    +

    If the new remote reference should always be encrypted then make sure to enable +encryption.

    +
    +
    +
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
+ -p password --create --xdcr-cluster-name east --xdcr-hostname 192.168.1.6 \
+ --xdcr-username Administrator --xdcr-password password \
+ --xdcr-demand-encryption 1 --xdcr-certificate /root/cluster.cert
    +
    +

    To list all current XDCR cluster references run the following command.

    +
    +
    +
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
+ -p password --list
    +
    +

    If you need to edit a cluster references named "east" and change the password +run the following command.

    +
    +
    +
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
+ -p password --edit --xdcr-cluster-name east --xdcr-hostname 192.168.1.6
+ --xdcr-username Administrator --xdcr-password new_password
    +
    +

    Note in the above example that you need to specify all of the current unchanging +configuration parameters also to avoid them being reset to defaults.

    +

    If you no longer need an XDCR remote reference then you can delete it. We should +this below using the "east" remote reference as an exmaple.

    +
    +
    +
    $ couchbase-cli xdcr-setup -c 192.168.1.5 -u Administrator \
+ -p password --delete --xdcr-cluster-name east
    +
    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    SEE ALSO

    +
    +

    couchbase-cli-setting-xdcr(1) +couchbase-cli-xdcr-replicate(1)

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcli/couchbase-cli.dita b/content/cli/cbcli/couchbase-cli.dita deleted file mode 100644 index 0c9759d3c4..0000000000 --- a/content/cli/cbcli/couchbase-cli.dita +++ /dev/null @@ -1,644 +0,0 @@ - - - - - couchbase-cli(1) - - - - - - - - - - - - - - - - - - - NAME - - - -

    couchbase-cli - The Couchbase cluster management utility.

    - - -     - - SYNOPSIS - - - - -couchbase-cli <command> [options] - - - - - - DESCRIPTION - - - -

    The couchbase-cli script is the main command line interface for Couchbase -cluster management.

    - - -     - - OPTIONS - - - -
    - -
    ---version -
    -
    -

    - Prints version information. -

    -
    -     - -
    --h,--help -
    -
    -

    - Prints the short and long help messages. -

    -
    -     -
    - - -     - - COMMANDS - - - -
    - -
    - -couchbase-cli-admin-role-manage1 - -
    -
    -

    - Manage LDAP user roles. -

    -
    -     - -
    - -couchbase-cli-bucket-compact1 - -
    -
    -

    - Compact a Couchbase data bucket. -

    -
    -     - -
    - -couchbase-cli-bucket-create1 - -
    -
    -

    - Create a new Couchbase data bucket. -

    -
    -     - -
    - -couchbase-cli-bucket-delete1 - -
    -
    -

    - Delete a Couchbase data bucket. -

    -
    -     - -
    - -couchbase-cli-bucket-edit1 - -
    -
    -

    - Edit a Couchbase data bucket. -

    -
    -     - -
    - -couchbase-cli-bucket-flush1 - -
    -
    -

    - Flush a Couchbase data bucket. -

    -
    -     - -
    - -couchbase-cli-bucket-list1 - -
    -
    -

    - List all Couchbase data buckets. -

    -
    -     - -
    - -couchbase-cli-cluster-edit1 - -
    -
    -

    - Edits cluster settings. -

    -
    -     - -
    - -couchbase-cli-cluster-init1 - -
    -
    -

    - Initializes a Couchbase cluster. -

    -
    -     - -
    - -couchbase-cli-collect-logs-start1 - -
    -
    -

    - Start log collection. -

    -
    -     - -
    - -couchbase-cli-collect-logs-status1 - -
    -
    -

    - Get log collection status. -

    -
    -     - -
    - -couchbase-cli-collect-logs-stop1 - -
    -
    -

    - Stop the current log collection task. -

    -
    -     - -
    - -couchbase-cli-failover1 - -
    -
    -

    - Failover a server in the cluster. -

    -
    -     - -
    - -couchbase-cli-group-manage1 - -
    -
    -

    - Manage server groups. -

    -
    -     - -
    - -couchbase-cli-host-list1 - -
    -
    -

    - Lists all hosts in the cluster. -

    -
    -     - -
    - -couchbase-cli-node-init1 - -
    -
    -

    - Initializes a node. -

    -
    -     - -
    - -couchbase-cli-rebalance1 - -
    -
    -

    - Rebalances data across nodes in a cluster. -

    -
    -     - -
    - -couchbase-cli-rebalance-status1 - -
    -
    -

    - Show the current rebalance status. -

    -
    -     - -
    - -couchbase-cli-rebalance-stop1 - -
    -
    -

    - Stops the current rebalance task. -

    -
    -     - -
    - -couchbase-cli-recovery1 - -
    -
    -

    - Recovers a previously failed over node. -

    -
    -     - -
    - -couchbase-cli-reset-admin-password1 - -
    -
    -

    - Resets the administrator password. -

    -
    -     - -
    - -couchbase-cli-server-add1 - -
    -
    -

    - Adds a server to the cluster. -

    -
    -     - -
    - -couchbase-cli-server-info1 - -
    -
    -

    - Displays server level information and statistics. -

    -
    -     - -
    - -couchbase-cli-server-list1 - -
    -
    -

    - Lists all servers in the cluster. -

    -
    -     - -
    - -couchbase-cli-server-readd1 - -
    -
    -

    - Adds a server back to the cluster after a failover. -

    -
    -     - -
    - -couchbase-cli-setting-alert1 - -
    -
    -

    - Modifies alert settings. -

    -
    -     - -
    - -couchbase-cli-setting-audit1 - -
    -
    -

    - Modifies audit log settings. -

    -
    -     - -
    - -couchbase-cli-setting-autofailover1 - -
    -
    -

    - Modifies auto-failover settings. -

    -
    -     - -
    - -couchbase-cli-setting-autoreprovision1 - -
    -
    -

    - Modifies auto-reprovision settings. -

    -
    -     - -
    - -couchbase-cli-setting-cluster1 - -
    -
    -

    - Modifies cluster settings. -

    -
    -     - -
    - -couchbase-cli-setting-compaction1 - -
    -
    -

    - Modifies compaction settings. -

    -
    -     - -
    - -couchbase-cli-setting-index1 - -
    -
    -

    - Modifies index settings. -

    -
    -     - -
    - -couchbase-cli-setting-ldap1 - -
    -
    -

    - Modifies LDAP settings. -

    -
    -     - -
    - -couchbase-cli-setting-notification1 - -
    -
    -

    - Modifies notification settings. -

    -
    -     - -
    - -couchbase-cli-setting-password-policy1 - -
    -
    -

    - Modifies the password policy. -

    -
    -     - -
    - -couchbase-cli-setting-security1 - -
    -
    -

    - Modifies security policies. -

    -
    -     - -
    - -couchbase-cli-setting-xdcr1 - -
    -
    -

    - Modifies cross datacenter replication (XDCR) settings. -

    -
    -     - -
    - -couchbase-cli-ssl-manage1 - -
    -
    -

    - Manage SSL certificates. -

    -
    -     - -
    - -couchbase-cli-user-manage1 - -
    -
    -

    - Manage RBAC users. -

    -
    -     - -
    - -couchbase-cli-xdcr-replicate1 - -
    -
    -

    - Manages XDCR cluster references. -

    -
    -     - -
    - -couchbase-cli-xdcr-setup1 - -
    -
    -

    - Manages XDCR replications. -

    -
    -     -
    - - -     - - HOST FORMATS - - - -

    When specifying a host for the couchbase-cli command the following formats are expected:

    -
      -
    • -

      -couchbase://<addr> -

      -
      • -
    • -

      -<addr>:<port> -

      -
      • -
    • -

      -http://<addr>:<port> -

      -
      • -
    -

    It is recommended to use the couchbase://<addr> format for standard -installations. The other two formats allow an option to take a port number which -is needed for non-default installations where the admin port has been set up on -a port other that 8091.

    - - -     - - ENVIRONMENT AND CONFIGURATION VARIABLES - - - -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line. It also allows the user to ensure that their - password are not cached in their command line history.

    -

    CB_REST_PASSWORD - Specifies the password of the user executing the command. This environment - variable allows you to specify a default argument for the -p/--password - argument on the command line.

    - - -     - - COUCHBASE-CLI - - - -

    Part of the -couchbase-cli1 - suite

    - - -     -     -     diff --git a/content/cli/cbcli/couchbase-cli.html b/content/cli/cbcli/couchbase-cli.html new file mode 100644 index 0000000000..5050f09506 --- /dev/null +++ b/content/cli/cbcli/couchbase-cli.html @@ -0,0 +1,1184 @@ + + + + + +couchbase-cli(1) + + + + + +
    +
    +

    SYNOPSIS

    +
    +
    +
    couchbase-cli <command> [options]
    +
    +
    +
    +
    +
    +

    DESCRIPTION

    +
    +

    The couchbase-cli script is the main command line interface for Couchbase +cluster management.

    +
    +
    +
    +

    OPTIONS

    +
    +
    +
    +--version +
    +
    +

    + Prints version information. +

    +
    +
    +-h,--help +
    +
    +

    + Prints the short and long help messages. +

    +
    +
    +
    +
    +
    +

    COMMANDS

    +
    +
    +
    +couchbase-cli-admin-role-manage(1) +
    +
    +

    + Manage LDAP user roles. +

    +
    +
    +couchbase-cli-bucket-compact(1) +
    +
    +

    + Compact a Couchbase data bucket. +

    +
    +
    +couchbase-cli-bucket-create(1) +
    +
    +

    + Create a new Couchbase data bucket. +

    +
    +
    +couchbase-cli-bucket-delete(1) +
    +
    +

    + Delete a Couchbase data bucket. +

    +
    +
    +couchbase-cli-bucket-edit(1) +
    +
    +

    + Edit a Couchbase data bucket. +

    +
    +
    +couchbase-cli-bucket-flush(1) +
    +
    +

    + Flush a Couchbase data bucket. +

    +
    +
    +couchbase-cli-bucket-list(1) +
    +
    +

    + List all Couchbase data buckets. +

    +
    +
    +couchbase-cli-cluster-edit(1) +
    +
    +

    + Edits cluster settings. +

    +
    +
    +couchbase-cli-cluster-init(1) +
    +
    +

    + Initializes a Couchbase cluster. +

    +
    +
    +couchbase-cli-collect-logs-start(1) +
    +
    +

    + Start log collection. +

    +
    +
    +couchbase-cli-collect-logs-status(1) +
    +
    +

    + Get log collection status. +

    +
    +
    +couchbase-cli-collect-logs-stop(1) +
    +
    +

    + Stop the current log collection task. +

    +
    +
    +couchbase-cli-eventing-function-setup(1) +
    +
    +

    + Manage the Eventing Service functions. +

    +
    +
    +couchbase-cli-failover(1) +
    +
    +

    + Failover a server in the cluster. +

    +
    +
    +couchbase-cli-group-manage(1) +
    +
    +

    + Manage server groups. +

    +
    +
    +couchbase-cli-host-list(1) +
    +
    +

    + Lists all hosts in the cluster. +

    +
    +
    +couchbase-cli-node-init(1) +
    +
    +

    + Initializes a node. +

    +
    +
    +couchbase-cli-rebalance(1) +
    +
    +

    + Rebalances data across nodes in a cluster. +

    +
    +
    +couchbase-cli-rebalance-status(1) +
    +
    +

    + Show the current rebalance status. +

    +
    +
    +couchbase-cli-rebalance-stop(1) +
    +
    +

    + Stops the current rebalance task. +

    +
    +
    +couchbase-cli-recovery(1) +
    +
    +

    + Recovers a previously failed over node. +

    +
    +
    +couchbase-cli-reset-admin-password(1) +
    +
    +

    + Resets the administrator password. +

    +
    +
    +couchbase-cli-server-add(1) +
    +
    +

    + Adds a server to the cluster. +

    +
    +
    +couchbase-cli-server-info(1) +
    +
    +

    + Displays server level information and statistics. +

    +
    +
    +couchbase-cli-server-list(1) +
    +
    +

    + Lists all servers in the cluster. +

    +
    +
    +couchbase-cli-server-readd(1) +
    +
    +

    + Adds a server back to the cluster after a failover. +

    +
    +
    +couchbase-cli-setting-alert(1) +
    +
    +

    + Modifies alert settings. +

    +
    +
    +couchbase-cli-setting-audit(1) +
    +
    +

    + Modifies audit log settings. +

    +
    +
    +couchbase-cli-setting-autofailover(1) +
    +
    +

    + Modifies auto-failover settings. +

    +
    +
    +couchbase-cli-setting-autoreprovision(1) +
    +
    +

    + Modifies auto-reprovision settings. +

    +
    +
    +couchbase-cli-setting-cluster(1) +
    +
    +

    + Modifies cluster settings. +

    +
    +
    +couchbase-cli-setting-compaction(1) +
    +
    +

    + Modifies compaction settings. +

    +
    +
    +couchbase-cli-setting-index(1) +
    +
    +

    + Modifies index settings. +

    +
    +
    +couchbase-cli-setting-ldap(1) +
    +
    +

    + Modifies LDAP settings. +

    +
    +
    +couchbase-cli-setting-notification(1) +
    +
    +

    + Modifies notification settings. +

    +
    +
    +couchbase-cli-setting-password-policy(1) +
    +
    +

    + Modifies the password policy. +

    +
    +
    +couchbase-cli-setting-security(1) +
    +
    +

    + Modifies security policies. +

    +
    +
    +couchbase-cli-setting-xdcr(1) +
    +
    +

    + Modifies cross datacenter replication (XDCR) settings. +

    +
    +
    +couchbase-cli-ssl-manage(1) +
    +
    +

    + Manage SSL certificates. +

    +
    +
    +couchbase-cli-user-manage(1) +
    +
    +

    + Manage RBAC users. +

    +
    +
    +couchbase-cli-xdcr-replicate(1) +
    +
    +

    + Manages XDCR cluster references. +

    +
    +
    +couchbase-cli-xdcr-setup(1) +
    +
    +

    + Manages XDCR replications. +

    +
    +
    +
    +
    +
    +

    HOST FORMATS

    +
    +

    When specifying a host for the couchbase-cli command the following formats are expected:

    +
      +
    • +

      +couchbase://<addr> +

      +
      • +
    • +

      +<addr>:<port> +

      +
      • +
    • +

      +http://<addr>:<port> +

      +
      • +
    +

    It is recommended to use the couchbase://<addr> format for standard +installations. The other two formats allow an option to take a port number which +is needed for non-default installations where the admin port has been set up on +a port other that 8091.

    +
    +
    +
    +

    ENVIRONMENT AND CONFIGURATION VARIABLES

    +
    +

    CB_REST_USERNAME + Specifies the username to use when executing the command. This environment + variable allows you to specify a default argument for the -u/--username + argument on the command line.

    +

    CB_REST_PASSWORD + Specifies the password of the user executing the command. This environment + variable allows you to specify a default argument for the -p/--password + argument on the command line. It also allows the user to ensure that their + password are not cached in their command line history.

    +
    +
    +
    +

    COUCHBASE-CLI

    +
    +

    Part of the couchbase-cli(1) suite

    +
    +
    +
    +
    + + + diff --git a/content/cli/cbcollect-info-tool.dita b/content/cli/cbcollect-info-tool.dita index e752c4cda9..87badedde3 100644 --- a/content/cli/cbcollect-info-tool.dita +++ b/content/cli/cbcollect-info-tool.dita @@ -1,34 +1,79 @@ - <cmdname>cbcollect_info</cmdname> - The cbcollect_info tool provides detailed statistics for a specific node. + + + <cmdname> + cbcollect_info + </cmdname> + + + + The cbcollect_info tool provides detailed statistics for a specific node. + + -
    Syntax -

    The basic syntax is:

    +
    + + Syntax + + +

    + The basic syntax is: +

    + cbcollect_info [options] [output-file] -

    where output-file is the name of the .zip file containing the diagnostic information.

    + +

    + where output-file is the name of the .zip file containing the diagnostic information. +

    +
    -
    Description + +
    + + Description + + +

    + The cbcollect_info command gathers statistics from an individual node in the + cluster. +

    -

    The cbcollect_info command gathers statistics from an individual node in the - cluster.

    -

    To collect diagnostic information for an entire cluster, run the command for every +

    + To collect diagnostic information for an entire cluster, run the command for every node that exists in that cluster. If you are experiencing problems with multiple - nodes in a cluster, you may need to run it on all nodes in a cluster.

    -

    A root account is required to run this command and collect all the server information. There + nodes in a cluster, you may need to run it on all nodes in a cluster. +

    + +

    + A root account is required to run this command and collect all the server information. There are internal server files and directories that this tool accesses that require root - privileges.

    -

    To use this command, remotely connect to the machine which contains the Couchbase Server then + privileges. +

    + +

    + To use this command, remotely connect to the machine which contains the Couchbase Server then issue the command with appropriate options. This command is typically run under the direction of technical support at Couchbase and generates a large .zip file. This - archive contains files which contain performance statistics and server logs.

    + archive contains files which contain performance statistics and server logs. +

    + cbcollect_info is one of the most important diagnostic tools. Run cbcollect_info on all nodes in the cluster, and upload - all of the resulting files to Couchbase technical support . -

    This tool is at the following locations:

    - Tool locations + all of the resulting files to Couchbase technical support. + + +

    + This tool is at the following locations: +

    + +
    + + Tool locations + + @@ -54,8 +99,15 @@
    -

    The tool creates the following log files in the output archive:

    - Output files + +

    + The tool creates the following log files in the output archive: +

    + +
    + Output files + + @@ -66,27 +118,56 @@ + + cbcollect_info.log + Contains information about the performance of cbcollect_info. + couchbase.log Contains system-level information, primarily the result of shell commands. Generated at collection-time. ddocs.log - Contains the design documents for each bucket in json format, taken from the view file on disk. + Contains the design documents for each bucket in JSON format, taken from the view file on disk. diag.log A diagnostic summary of the cluster manager, obtained by downloading /diag output from the node's running service. + + kv_trace.json + Metadata used for Phosphor KV tracing. + ini.log A dump of all ini configuration files. + + master_events.log + Contains major events that have affected the cluster, like bucket creation, and rebalance + information such as vBucket movement. + memcached.log - Contains information relating to the core memcached component, + Contains information relating to the core memcached component, including DCP stream requests and slow operations. + + ns_server.analytics.log + (Developer Preview.) + + + ns_server.analytics_debug.log + (Developer Preview.) + + + ns_server.analytics_shutdown.log + (Developer Preview.) + + + ns_server.analytics_trace.log + (Developer Preview.) + ns_server.babysitter.log Troubleshooting log for the babysitter process which is responsible for spawning all @@ -96,7 +177,7 @@ ns_server.couchdb.log Troubleshooting log for the couchdb subsystem which underlies - map-reduce and spatial views + map-reduce and spatial views. ns_server.debug.log @@ -107,6 +188,10 @@ ns_server.error.log Error-level troubleshooting log for the cluster management component. + + ns_server.eventing.log + Eventing service logs. + ns_server.fts.log Troubleshooting logs for the full-text search service. @@ -135,6 +220,10 @@ ns_server.info.log Info-level troubleshooting log for the cluster management component. + + ns_server.json_rpc.log + JSON RPC logs. + ns_server.mapreduce_errors.log JavaScript and other view-processing errors are reported in this @@ -153,6 +242,10 @@ Troubleshooting log for the projector process which is responsible for sending appropriate mutations from Data nodes to Index nodes. + + ns_server.query.log + Query service logs. + ns_server.reports.log Contains progress and crash reports for the Erlang processes. Due to the nature of @@ -178,6 +271,10 @@ ns_server.xdcr_errors.log Error-level troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions prior to 4.0. + + ns_server.xdcr_target.log + Log of traces from cross datacenter replication (XDCR) target endpoint handlers. + ns_server.xcdr_trace.log Trace-level troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions prior to 4.0. @@ -195,6 +292,10 @@ syslog.tar.gz Archive of various system-level logs. + + systemd_journal.gz + (Linux only.) The logs from systemd. +
    @@ -205,7 +306,7 @@ cbcollect_info options - + @@ -214,62 +315,98 @@ + -h, --help Shows help information. + -r ROOT - Root directory. Default to - /opt/couchbase/bin/.. + Specifies the root directory. Defaults to the parent directory of the directory containing + cbcollect_info, e.g. /opt/couchbase/bin/.. on Linux. + -v - Increases verbosity level. If specified, debugging information - for cbcollect_info is also displayed on your - console. + Increases the verbosity level: If specified, debugging information + will be included in the console display. + -p - Gathers only product-related information. + Specifies that only product-related information should be gathered. + -d - Dumps a list of commands that cbcollect_info - needs. + Dumps a list of commands required by cbcollect_info. + --bypass-sensitive-data - Skips collecting breakpad crash dumps if set to true. + If set to true, skips collecting breakpad crash-dumps. + + + + --task-regexp=TASK_REGEXP + Runs only those tasks that match the specified regular expression. (This flag is provided + for debugging purposes.) + + + + --tmp-dir=TMP_DIR + Specifies the tmp directory that is used during data processing. This setting overrides any + existing setting of the TMPDIR environment variable. + - --initargs-INITARG - The server initargs path. + --initargs=INITARGS + Specifies the server initargs path. + - --single-node-diag - Collects on a per node basis, diagnostics on just this node. The - default is all reachable nodes. + --multi-node-diag + Specifies that diagnostic information should be collected for each reachable + node in the cluster. The + default is for collection on the current node only. + + + --log-redaction-level=REDACT_LEVEL + Specifies the redaction level for the logs collected. The level can be either + none (the default) or partial. + + + + --log-redaction-salt=SALT_VALUE + Salts the hashing of tagged data. The default value is a random uuid. If this + flag is used, a --log-redaction-level value should be specified. + + --upload-host=UPLOAD_HOST - Fully-qualified domain name of the host you want the logs uploaded to. The protocol + Specifies the fully-qualified domain name of the host you want the logs uploaded to. The protocol prefix of the domain name, https://, is optional. It is the default-only supported protocol. --customer=UPLOAD_CUSTOMER - Customer name. This value is a string with a maximum length of - 50 characters that contains only the following characters: - [A-Za-z0-9_.-]. If any other characters are included in the string, - the request is rejected. + Specifies the customer name. This value must be a string whose maximum length is + 50 characters. Only the following characters can be used: + [A-Za-z0-9_.-]. If any other characters are included, + the request is rejected. + + + + --upload-proxy=UPLOAD_PROXY + Specifies a proxy for upload. + --ticket=UPLOAD_TICKET - Couchbase support ticket number. This value is a string with a - maximum length of 7 character that contains only digits 0-9. The - ticket number is optional and is used only in conjunction with the - --upload option. + Specifies the Couchbase Support ticket-number. The value must be a string with a + maximum length of 7 characters, containing only digits in the range of 0-9. @@ -288,41 +425,61 @@
    Examples -

    To create a diagnostics .zip file, log onto the node and run the cbcollect_info tool.

    +

    To create a diagnostics .zip file, log onto the node and run the cbcollect_info tool.

    On Linux, run as root or use sudo: sudo /opt/couchbase/bin/cbcollect_info <node_name>.zip

    On Windows, run as Administrator: C:\Program Files\Couchbase\Server\bin\cbcollect_info <node_name>.zip

    Response

    -

    The following example response shows partial output when running the cbcollect_info command.

    - uname (uname -a) - OK +

    The following example response shows partial output when running the cbcollect_info command.

    + Using temporary dir /tmp +Log Redaction () - OK +uname (uname -a) - OK time and TZ (date; date -u) - OK -raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK -raw /etc/timezone (cat /etc/timezone) - Exit code 1 -System Hardware (lshw -json || lshw) - Exit code 127 +ntp time (ntpdate -q pool.ntp.org || nc time.nist.gov 13 || netcat time.nist.gov 13) - OK +ntp peers (ntpq -p) - Exit code 127 +raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - Exit code 1 +raw /etc/timezone (cat /etc/timezone) - OK +System Hardware (lshw -json || lshw) - OK +Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK ... -adding: /tmp/tmpMYbSyD/couchbase.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/couchbase.log -adding: /tmp/tmpMYbSyD/ns_server.xdcr.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.xdcr.log -adding: /tmp/tmpMYbSyD/ns_server.couchdb.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.couchdb.log -adding: /tmp/tmpMYbSyD/stats.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/stats.log -adding: /tmp/tmpMYbSyD/ini.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ini.log -adding: /tmp/tmpMYbSyD/ns_server.error.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.error.log -adding: /tmp/tmpMYbSyD/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.ssl_proxy.log -adding: /tmp/tmpMYbSyD/ns_server.views.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.views.log -adding: /tmp/tmpMYbSyD/ns_server.info.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.info.log -adding: /tmp/tmpMYbSyD/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.xdcr_errors.log -adding: /tmp/tmpMYbSyD/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.mapreduce_errors.log -adding: /tmp/tmpMYbSyD/diag.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/diag.log -adding: /tmp/tmpMYbSyD/ns_server.http_access.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.http_access.log -adding: /tmp/tmpMYbSyD/syslog.tar.gz -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/syslog.tar.gz -adding: /tmp/tmpMYbSyD/ns_server.debug.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.debug.log -adding: /tmp/tmpMYbSyD/ddocs.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ddocs.log -adding: /tmp/tmpMYbSyD/ns_server.reports.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.reports.log -adding: /tmp/tmpMYbSyD/memcached.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/memcached.log -adding: /tmp/tmpMYbSyD/ns_server.babysitter.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.babysitter.log -adding: /tmp/tmpMYbSyD/ns_server.stats.log -> cbcollect_info_ns_1@10.5.2.117_20141209-024045/ns_server.stats.log +adding: /tmp/tmpY7p_3T/couchbase.log -> cbcollect_info__20180718-084858/couchbase.log +adding: /tmp/tmpY7p_3T/ns_server.error.log -> cbcollect_info__20180718-084858/ns_server.error.log +adding: /tmp/tmpY7p_3T/ns_server.analytics_debug.log -> cbcollect_info__20180718-084858/ns_server.analytics_debug.log +adding: /tmp/tmpY7p_3T/ns_server.json_rpc.log -> cbcollect_info__20180718-084858/ns_server.json_rpc.log +adding: /tmp/tmpY7p_3T/cbcollect_info.log -> cbcollect_info__20180718-084858/cbcollect_info.log +adding: /tmp/tmpY7p_3T/ns_server.ns_couchdb.log -> cbcollect_info__20180718-084858/ns_server.ns_couchdb.log +adding: /tmp/tmpY7p_3T/ns_server.analytics.log -> cbcollect_info__20180718-084858/ns_server.analytics.log +adding: /tmp/tmpY7p_3T/stats.log -> cbcollect_info__20180718-084858/stats.log +adding: /tmp/tmpY7p_3T/memcached.log -> cbcollect_info__20180718-084858/memcached.log +adding: /tmp/tmpY7p_3T/ns_server.indexer.log -> cbcollect_info__20180718-084858/ns_server.indexer.log +adding: /tmp/tmpY7p_3T/ns_server.fts.log -> cbcollect_info__20180718-084858/ns_server.fts.log +adding: /tmp/tmpY7p_3T/ns_server.http_access_internal.log -> cbcollect_info__20180718-084858/ns_server.http_access_internal.log +adding: /tmp/tmpY7p_3T/ns_server.xdcr_target.log -> cbcollect_info__20180718-084858/ns_server.xdcr_target.log +adding: /tmp/tmpY7p_3T/ns_server.eventing.log -> cbcollect_info__20180718-084858/ns_server.eventing.log +adding: /tmp/tmpY7p_3T/syslog.tar.gz -> cbcollect_info__20180718-084858/syslog.tar.gz +adding: /tmp/tmpY7p_3T/ns_server.reports.log -> cbcollect_info__20180718-084858/ns_server.reports.log +adding: /tmp/tmpY7p_3T/ns_server.query.log -> cbcollect_info__20180718-084858/ns_server.query.log +adding: /tmp/tmpY7p_3T/ns_server.couchdb.log -> cbcollect_info__20180718-084858/ns_server.couchdb.log +adding: /tmp/tmpY7p_3T/ns_server.goxdcr.log -> cbcollect_info__20180718-084858/ns_server.goxdcr.log +adding: /tmp/tmpY7p_3T/master_events.log -> cbcollect_info__20180718-084858/master_events.log +adding: /tmp/tmpY7p_3T/ns_server.info.log -> cbcollect_info__20180718-084858/ns_server.info.log +adding: /tmp/tmpY7p_3T/diag.log -> cbcollect_info__20180718-084858/diag.log +adding: /tmp/tmpY7p_3T/ns_server.analytics_shutdown.log -> cbcollect_info__20180718-084858/ns_server.analytics_shutdown.log +adding: /tmp/tmpY7p_3T/systemd_journal.gz -> cbcollect_info__20180718-084858/systemd_journal.gz +adding: /tmp/tmpY7p_3T/ns_server.analytics_trace.json -> cbcollect_info__20180718-084858/ns_server.analytics_trace.json +adding: /tmp/tmpY7p_3T/ns_server.stats.log -> cbcollect_info__20180718-084858/ns_server.stats.log +adding: /tmp/tmpY7p_3T/ns_server.views.log -> cbcollect_info__20180718-084858/ns_server.views.log +adding: /tmp/tmpY7p_3T/ini.log -> cbcollect_info__20180718-084858/ini.log +adding: /tmp/tmpY7p_3T/kv_trace.json -> cbcollect_info__20180718-084858/kv_trace.json +adding: /tmp/tmpY7p_3T/ns_server.mapreduce_errors.log -> cbcollect_info__20180718-084858/ns_server.mapreduce_errors.log +adding: /tmp/tmpY7p_3T/ns_server.projector.log -> cbcollect_info__20180718-084858/ns_server.projector.log +adding: /tmp/tmpY7p_3T/ns_server.metakv.log -> cbcollect_info__20180718-084858/ns_server.metakv.log +adding: /tmp/tmpY7p_3T/ns_server.debug.log -> cbcollect_info__20180718-084858/ns_server.debug.log +adding: /tmp/tmpY7p_3T/ns_server.babysitter.log -> cbcollect_info__20180718-084858/ns_server.babysitter.log +adding: /tmp/tmpY7p_3T/ns_server.http_access.log -> cbcollect_info__20180718-084858/ns_server.http_access.log
    diff --git a/content/cli/cbepctl-intro.dita b/content/cli/cbepctl-intro.dita index 07aa9f168b..12e68b7be0 100644 --- a/content/cli/cbepctl-intro.dita +++ b/content/cli/cbepctl-intro.dita @@ -54,25 +54,21 @@ -
    Options +
    Syntax

    Basic syntax for this command:

    - cbepctl [host]:11210 -b [bucket_name] -p [bucket-password] start - cbepctl [host]:11210 -b [bucket_name] -p [bucket-password] stop - cbepctl [host]:11210 -b [bucket_name] -p [bucket-password] set [command-type] [parameter] [value] - + cbepctl [host]:11210 [command] [options]

    For this command, host is the IP address for the Couchbase Server cluster or node in the cluster. The port is always the standard port used for cluster-wide stats and is - at 11210. The bucket name and password is provided first and then - the command option and authentication.

    -

    Use the following command options to manage persistence:

    + at 11210.

    +

    Use the following commands to manage persistence:

    - cbepctl command options + cbepctl commands - Command option + Command Description @@ -91,21 +87,64 @@ Wait until queues are drained - set + set [type] [param] [value] Set checkpoint_param, flush_param, and tap_param command types. This changes how or - when the persisting occurs. + when persistence occurs.
    -

    The set command option along with the checkpoint_param, +

    The set command along with the checkpoint_param, flush_param, and tap_param command types - change the behavior of persistence in Couchbase Server.

    + changes the behavior of persistence in Couchbase Server.

    +
    Options + +

    The following are the command options:

    + + + + + + + + Option + Description + + + + + -h, --help + Shows the help message and exits. + + + -a + Iterates over all buckets. + + + -b [BUCKETNAME] + The bucket to control. Default: default. + + + -u [USERNAME] + The username to authenticate. Default: the bucket name. + + + -p [PASSWORD] + The password for the bucket, if one exists. + + + -S + Reads the password from stdin. + + + +
    +
    diff --git a/content/cli/cbepctl/set-flush_param.dita b/content/cli/cbepctl/set-flush_param.dita index eaae03c9fc..bbde9dc157 100644 --- a/content/cli/cbepctl/set-flush_param.dita +++ b/content/cli/cbepctl/set-flush_param.dita @@ -58,7 +58,7 @@ set flush_param mutation_mem_threshold [value]
    alog_sleep_time, alog_task_time
    Couchbase Server has an optimized disk + href="../../understanding-couchbase/buckets-memory-and-storage/memory.dita#memory/initialization-and-warmup">disk warmup. Couchbase Server pre-fetches a list of most-frequently accessed keys and fetches these documents first. The server runs a periodic scanner process which determines which keys are most frequently used. The cbepctl flush_param @@ -77,7 +77,7 @@ set flush_param mutation_mem_threshold [value]
    exp_pager_stime
    The cbepctl flush_param exp_pager_stime command sets the time interval for disk cleanup. Couchbase Server does lazy expiration, that + href="../../understanding-couchbase/buckets-memory-and-storage/memory.dita#memory/expiry-pager">expiration, that is, expired items are flagged as deleted rather than being immediately erased. Couchbase Server has a maintenance process that periodically looks through all information and erases expired items. By default, this maintenance process runs every 60 minutes, but it @@ -90,7 +90,7 @@ set flush_param mutation_mem_threshold [value]
    mem_low_wat, mem_high_wat, pager_active_vb_pcnt
    -
    Ejection means that documents are removed from RAM but the key and metadata remain. If the amount of RAM used by items reaches the high water mark (upper threshold), both active and replica data are ejected until the memory usage (amount of diff --git a/content/cli/cbrecovery.dita b/content/cli/cbrecovery.dita new file mode 100644 index 0000000000..2fba399de5 --- /dev/null +++ b/content/cli/cbrecovery.dita @@ -0,0 +1,344 @@ + + + + + + <cmdname> + cbrecovery + </cmdname> + + + + The cbrecovery tool restores data to a local cluster, from a bucket + on a remote cluster that was previously established as an XDCR remote replica. + + + + + +
    + + Syntax + + +

    + The basic syntax is: +

    + cbrecovery [options] [source-cluster] [destination-cluster] + +

    + Where: +

    + +
      + +
    • + [options]: Command options for cbrecovery. +
      • + +
    • + [source-cluster]: A cluster containing an XDCR remote-replica bucket, which is + used as the source for data-recovery. +
      • + +
    • + [destination-cluster]: A cluster that has suffered data loss from a bucket + previously replicated to a remote cluster, by means of XDCR. +
      • + +
    + +
    + +
    + + Description + + +

    + Couchbase Server allows one or more replicas to be created for each vBucket on the cluster. This helps + to ensure continued data-availability in the event of node-failure. +

    + +

    + However, if multiple nodes within a single cluster fail simultaneously, + one or more active vBuckets and all their replicas may be affected; meaning that lost data cannot be + recovered locally. +

    + +

    + In such cases, + provided that a bucket affected by such failure has already been established as a source bucket for XDCR, the + lost data may be retrieved from the bucket defined + on the remote server as the corresponding replication-target. This retrieval is achieved from the command-line, by means of + cbrecovery. +

    + +

    + Before attempting to recover the lost data, restore capacity to the local cluster, + as appropriate. However, do not rebalance added nodes into the cluster until after + cbrecovery has been used to recover the lost data. +

    + + Tool Location

    The tool is in the following locations:

    + + + + + + Operating system + Location + + + + + Linux + /opt/couchbase/bin/cbrecovery + + + Windows + C:\Program Files\Couchbase\Server\bin\cbrecovery + + + Mac OS X + /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbrecovery + + + +
    +
    + +
    Options + + +

    The following are the command options:

    + + cbrestore options + + + + + + Option + Description + + + + + -h, --help + Command line help. + + + + -b BUCKET_SOURCE, --bucket-source=BUCKET_SOURCE + Name of a bucket on the source cluster. This must previously have been + established and used as a replication target for XDCR, using the source-bucket + specified here as BUCKET_DESTINATION. + + + + + -B BUCKET_DESTINATION, + --bucket-destination=BUCKET_DESTINATION + Name of a bucket on the destination cluster. This is a bucket that has suffered + data-loss, and was previously established and used as a source for XDCR replication, using + the target-bucket specified here as BUCKET_SOURCE. + + + + -u USERNAME, --username=USERNAME + The username for the Full Administrator, Cluster Administrator, or XDCR Administrator + for the cluster from which the data is recovered. + + + + -p PASSWORD, --password=PASSWORD + The password for the Full Administrator, Cluster Administrator, or XDCR Administrator + for the cluster from which the data is recovered. + + + + -U USERNAME_DESTINATION, --username-destination=USERNAME_DESTINATION + The username for the Full Administrator, Cluster Administrator, or XDCR Administrator + for the cluster onto which the recovered data is copied. + + + + -P PASSWORD_DESTINATION, --password-destination=PASSWORD_DESTINATION + The password for the Full Administrator, Cluster Administrator, or XDCR Administrator + for the cluster onto which the recovered data is copied. + + + + -i ID, --id=ID + Transfer only items that match a vBucket ID. + + + + -s, --ssl + Transfer data with SSL enabled. + + + + -t THREADS, --threads=THREADS + Number of concurrent worker-threads, performing the transfer. + + + + + -n, --dry-run + No actual transfer. Just validate parameters, files, connectivity, and + configurations. + + + + + -v, --verbose + Verbose logging. More v's provide more verbosity. Max: -vvv. + + + + -x EXTRA, --extra=EXTRA + Provide extra, uncommon configuration-parameters. Comma-separated + key=val(key-val)* pairs. + + + +
    + + + +

    + The following are extra, specialized command options, used by means of the cbrecovery -x + parameter. +

    + + + cbrestore -x options + + + + + + -x option + Description + + + + + backoff_cap=10 + Maximum back-off time during the rebalance period. + + + batch_max_bytes=400000 + Transfer this # of bytes per batch. + + + batch_max_size=1000 + Transfer this # of documents per batch. + + + cbb_max_mb=100000 + Split backup file on destination cluster if it exceeds the MB. + + + conflict_resolve=1 + By default, enable conflict resolution. + + + data_only=0 + For value 1, transfer only data from a backup file or cluster. + + + + dcp_consumer_queue_length=1000 + A DCP client + needs a queue for incoming documents/messages. A large length is more + efficient, but memory proportional to length*avg. doc size. Below + length 150, performance degrades significantly. + + + + design_doc_only=0 + For value 1, transfer only design documents from + a backup file or cluster. Default: 0. + The design documents are restored from a backup file created with + the cbbackup tool. + + + + max_retry=10 + Max number of sequential retries if the transfer fails. + + + + mcd_compatible=1 + For value 0, display extended fields for stdout output. + + + + nmv_retry=1 + 0 or 1, where 1 retries transfer after a NOT_MY_VBUCKET message. Default: + 1. + + + + recv_min_bytes=4096 + Amount of bytes for every TCP/IP batch transferred. + + + + rehash=0 + For value 1, rehash the partition IDs of each item as it is required when + transferring data between clusters with the different number of partitions; for + example, when transferring data from an Mac OS X server to a non-Mac OS X + cluster. + + + report=5 + Number batches transferred before updating the progress bar in the + console. + + + + report_full=2000 + Number batches transferred before emitting the progress information in the + console. + + + + seqno=0 + By default, start seqno from beginning. + + + + try_xwm=1 + Transfer documents with metadata. Default: 1. The value of 0 is used only when + transferring from 1.8.x to 1.8.x. + + + + +
    +
    + +
    + + Example + + + $ cbrecovery http://10.142.180.104:8091 http://10.142.180.101:8091 \ + > -b travelSampleBackup \ + > -B travel-sample \ + > -u Administrator \ + > -p password \ + > -U Administrator \ + > -P password \ + > -v + +
    + + +     + +     diff --git a/content/cli/cbrestore-tool.dita b/content/cli/cbrestore-tool.dita index ccdcf5f5fb..56bf992f01 100644 --- a/content/cli/cbrestore-tool.dita +++ b/content/cli/cbrestore-tool.dita @@ -65,8 +65,8 @@

    By default, when restoring from a backup using cbrestore, Couchbase Server will perform conflict resolution on all documents to be restored. The conflict resolution behavior is the same as cross datacenter replication (XDCR), which is detailed in - XDCR - architecture. This is so that documents which have been updated after the backup + Availability. + This is so that documents which have been updated after the backup are not incorrectly overwritten by the restore process.

    As a result, after running the restore process, some documents may not match the content of the backup file. In certain cases where a document has been recently deleted on the cluster, the document may not be @@ -261,9 +261,10 @@ design_doc_only=0 - The documents are restored from a backup file (created with the - cbbackup) tool. For value 1, transfer only design documents from - a backup file or cluster. Default: 0. + For value 1, transfer only design documents from + a backup file or cluster. Default: 0. +

    The design documents are restored from a backup file created with + the cbbackup tool.

    max_retry=10 @@ -310,7 +311,10 @@ uncompress=0 - For value 1, restore data in uncompressed mode. + For value 1, restore data in uncompressed mode. +

    This option is unsupported. To restore from compressed backups, you should use + , which is available for + Couchbase Server Enterprise Edition only.

         diff --git a/content/cli/cbrestore-tween-opsys.dita b/content/cli/cbrestore-tween-opsys.dita index 6688a9ae69..b38e96a8d1 100644 --- a/content/cli/cbrestore-tween-opsys.dita +++ b/content/cli/cbrestore-tween-opsys.dita @@ -3,7 +3,7 @@ <codeph>rehash=1</codeph> Data is restored to an operating system that is different from the backup operating - system using the cbrestore tool with the restore=1 + system using the cbrestore tool with the rehash=1 option.
    Syntax diff --git a/content/cli/cbstats-intro.dita b/content/cli/cbstats-intro.dita index e3ab2cd444..d1bed8f5d1 100644 --- a/content/cli/cbstats-intro.dita +++ b/content/cli/cbstats-intro.dita @@ -9,10 +9,11 @@
    Syntax

    The basic syntax is:

    - cbstats [host]: port [command] [options] + cbstats host:port [common options] command [command options]
    Description -

    The cbstats tool queries differently depending on the port:

      +

      The cbstats tool queries differently depending on the port:

      +
      • When using the Couchbase data port 11210, it will give you operations per node per bucket: /opt/couchbase/bin/cbstats -b beer-sample localhost:11210 all | grep \ curr_items: @@ -21,7 +22,8 @@ curr_items:                        provide cluster-wide stats: /opt/couchbase/bin/cbstats -b beer-sample localhost:11211 all | grep \ curr_items: curr_items:                                 7303
        • -
      If a named bucket is not provided, the server applies the setting to any +
    +

    If a named bucket is not provided, the server applies the setting to any default bucket that exists at the specified node.

    The tool is found in the @@ -173,19 +175,19 @@ curr_items:                        ep_vbucket_del - Number of vbucket deletion events. + Number of vBucket deletion events. ep_vbucket_del_fail - Number of failed vbucket deletion events. + Number of failed vBucket deletion events. ep_vbucket_del_max_walltime - Max wall time (µs) spent by deleting a vbucket. + Max wall time (µs) spent by deleting a vBucket. ep_vbucket_del_avg_walltime - Avg wall time (µs) spent by deleting a vbucket. + Avg wall time (µs) spent by deleting a vBucket. ep_flush_duration_total @@ -241,7 +243,7 @@ curr_items:                        ep_kv_size Memory used to store item metadata, keys and values, no matter - the vbucket’s state. If an item’s value is ejected, this stats will + the vBucket’s state. If an item’s value is ejected, this stats will be decremented by the size of the item’s value. @@ -263,7 +265,7 @@ curr_items:                        ep_total_cache_size - The total byte size of all items, no matter the vbucket’s state, + The total byte size of all items, no matter the vBucket’s state, no matter if an item’s value is ejected. @@ -410,7 +412,7 @@ curr_items:                        ep_pending_ops_max - Max ops seen awaiting 1 pending vbucket. + Max ops seen awaiting 1 pending vBucket. ep_pending_ops_max_duration @@ -505,7 +507,7 @@ curr_items:                        ep_chk_persistence_timeout - Timeout for vbucket checkpoint persistence. + Timeout for vBucket checkpoint persistence. ep_chk_remover_stime @@ -584,13 +586,13 @@ curr_items:                        ep_keep_closed_chks - True if we want to keep the closed checkpoints for each vbucket + True if we want to keep the closed checkpoints for each vBucket unless the memory usage is above high water mark. ep_max_checkpoints The maximum amount of checkpoints that can be in memory per - vbucket. + vBucket. ep_max_item_size @@ -687,7 +689,7 @@ curr_items:                        ep_vb0 - Whether vbucket 0 should be created by default. + Whether vBucket 0 should be created by default. ep_waitforwarmup @@ -734,7 +736,7 @@ curr_items:                        bucket you are interested in. If you don't specify a bucket (-b XXX), then you will get information for the default bucket, if it exists. If the default bucket does not exist, the - output will be empty.

    The following table shows replica vbucket statistics.

    + output will be empty.

    The following table shows replica vBucket statistics.

    @@ -824,7 +826,7 @@ curr_items:                       
    -

    The following table shows active vbucket statistics:

    +

    The following table shows active vBucket statistics:

    @@ -914,7 +916,7 @@ curr_items:                       
    -

    The following table shows pending vbucket statistics:

    +

    The following table shows pending vBucket statistics:

    @@ -1004,15 +1006,14 @@ curr_items:                       
    -
    - -
    Options -

    The majority of cbstats commands are predominantly used by Couchbase internally and to help resolve customer support incidents.

    +
    + +
    Options

    The following are the command options:

    @@ -1031,18 +1032,34 @@ curr_items:                        -h, --help Shows the help message and exits. + + -a + Iterates over all buckets. + -j - Outputs the result in JSON format + Outputs the result in JSON format. + + + -8 + Forces the use of UTF8 symbols in the output. -b [BUCKETNAME] - The bucket to get the status from. Default: default. + The bucket to get the statistics from. Default: default. + + + -u [USERNAME] + The username to authenticate. Default: the bucket name. -p [PASSWORD] The password for the bucket, if one exists. + + -S + Reads the password from stdin. + @@ -1052,27 +1069,27 @@ curr_items:                       
    Examples -

    Example with two buckets, default and c: - $ cbstats localhost:12000 -b default uuid +

    Example with two buckets, default and c:

    + $ cbstats localhost:11210 -b default uuid uuid: 3cd3d2efb20f3b9d7d484925b6d8c03f - $ cbstats localhost:12000 uuid + $ cbstats localhost:11210 uuid uuid: 3cd3d2efb20f3b9d7d484925b6d8c03f - $ cbstats localhost:12000 -b c uuid - uuid: 1de74b8dc0981abbc6481c5276b91be1

    + $ cbstats localhost:11210 -b c uuid + uuid: 1de74b8dc0981abbc6481c5276b91be1

    Example after the default bucket was deleted, and only the c bucket was - left: - $ cbstats localhost:12000 -b default uuid + left:

    + $ cbstats localhost:11210 -b default uuid Authentication error for default - $ cbstats localhost:12000 uuid + $ cbstats localhost:11210 uuid <nothing> - $ cbstats localhost:12000 -b c uuid - uuid: 1de74b8dc0981abbc6481c5276b91be1

    + $ cbstats localhost:11210 -b c uuid + uuid: 1de74b8dc0981abbc6481c5276b91be1

    Example for getting statistics for timings:

    To get statistics, for example, timings on host 10.5.2.117:

    - cbstats 10.5.2.117:11210 timings + cbstats 10.5.2.117:11210 -u Administrator -p password -b beer-sample timings

    Response:

    diff --git a/content/cli/cbstats/cbstats-allocator.dita b/content/cli/cbstats/cbstats-allocator.dita index 7b9fb60eaf..27299faff6 100644 --- a/content/cli/cbstats/cbstats-allocator.dita +++ b/content/cli/cbstats/cbstats-allocator.dita @@ -7,111 +7,135 @@
    Syntax

    Request syntax:

    - cbstats [host]:11210 allocator + cbstats host:11210 [common options] allocator
    Description

    This command is used to collect information on memory management.

    Options -

    None

    +

    There are no options for this command. For common cbstats options, see .

    Examples -

    Example request:

    - cbstats 10.5.2.54:11210 allocator +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample allocator -

    Response:

    - ------------------------------------------------- -MALLOC: 211296632 ( 201.5 MiB) Bytes in use by application -MALLOC: + 11730944 ( 11.2 MiB) Bytes in page heap freelist -MALLOC: + 10833696 ( 10.3 MiB) Bytes in central cache freelist -MALLOC: + 3404400 ( 3.2 MiB) Bytes in transfer cache freelist -MALLOC: + 18832632 ( 18.0 MiB) Bytes in thread cache freelists -MALLOC: + 1781920 ( 1.7 MiB) Bytes in malloc metadata -MALLOC: ------------ -MALLOC: = 257880224 ( 245.9 MiB) Actual memory used (physical + swap) -MALLOC: + 1236992 ( 1.2 MiB) Bytes released to OS (aka unmapped) -MALLOC: ------------ -MALLOC: = 259117216 ( 247.1 MiB) Virtual address space used -MALLOC: -MALLOC: 8855 Spans in use -MALLOC: 16 Thread heaps in use -MALLOC: 8192 Tcmalloc page size ------------------------------------------------- -Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). -Bytes released to the OS take up virtual address space but no physical memory. ------------------------------------------------- -Total size of freelists for per-thread caches, -transfer cache, and central cache, by size class ------------------------------------------------- -class 1 [ 8 bytes ] : 7825 objs; 0.1 MiB; 0.1 cum MiB -class 2 [ 16 bytes ] : 1149 objs; 0.0 MiB; 0.1 cum MiB -class 3 [ 32 bytes ] : 47076 objs; 1.4 MiB; 1.5 cum MiB -class 4 [ 48 bytes ] : 51137 objs; 2.3 MiB; 3.9 cum MiB -class 5 [ 64 bytes ] : 3987 objs; 0.2 MiB; 4.1 cum MiB -class 6 [ 80 bytes ] : 5528 objs; 0.4 MiB; 4.5 cum MiB -class 7 [ 96 bytes ] : 18530 objs; 1.7 MiB; 6.2 cum MiB -class 8 [ 112 bytes ] : 39207 objs; 4.2 MiB; 10.4 cum MiB -class 9 [ 128 bytes ] : 25812 objs; 3.2 MiB; 13.6 cum MiB -class 10 [ 144 bytes ] : 399 objs; 0.1 MiB; 13.6 cum MiB -class 11 [ 160 bytes ] : 321 objs; 0.0 MiB; 13.7 cum MiB -class 12 [ 176 bytes ] : 201 objs; 0.0 MiB; 13.7 cum MiB -class 13 [ 192 bytes ] : 162 objs; 0.0 MiB; 13.7 cum MiB -class 14 [ 208 bytes ] : 70 objs; 0.0 MiB; 13.7 cum MiB -class 15 [ 224 bytes ] : 77 objs; 0.0 MiB; 13.8 cum MiB -class 16 [ 240 bytes ] : 72 objs; 0.0 MiB; 13.8 cum MiB -class 17 [ 256 bytes ] : 81 objs; 0.0 MiB; 13.8 cum MiB -class 18 [ 288 bytes ] : 38 objs; 0.0 MiB; 13.8 cum MiB -class 19 [ 320 bytes ] : 52 objs; 0.0 MiB; 13.8 cum MiB -class 20 [ 352 bytes ] : 112 objs; 0.0 MiB; 13.9 cum MiB -class 21 [ 384 bytes ] : 44 objs; 0.0 MiB; 13.9 cum MiB -class 22 [ 416 bytes ] : 18 objs; 0.0 MiB; 13.9 cum MiB -class 23 [ 448 bytes ] : 60 objs; 0.0 MiB; 13.9 cum MiB -class 24 [ 480 bytes ] : 77 objs; 0.0 MiB; 13.9 cum MiB -class 25 [ 512 bytes ] : 2726 objs; 1.3 MiB; 15.3 cum MiB -class 26 [ 576 bytes ] : 114 objs; 0.1 MiB; 15.3 cum MiB -class 27 [ 640 bytes ] : 84 objs; 0.1 MiB; 15.4 cum MiB -class 28 [ 704 bytes ] : 99 objs; 0.1 MiB; 15.4 cum MiB -class 29 [ 768 bytes ] : 125 objs; 0.1 MiB; 15.5 cum MiB -class 30 [ 832 bytes ] : 124 objs; 0.1 MiB; 15.6 cum MiB -class 31 [ 896 bytes ] : 43 objs; 0.0 MiB; 15.7 cum MiB -class 32 [ 960 bytes ] : 24 objs; 0.0 MiB; 15.7 cum MiB -class 33 [ 1024 bytes ] : 45 objs; 0.0 MiB; 15.7 cum MiB -class 34 [ 1152 bytes ] : 29 objs; 0.0 MiB; 15.8 cum MiB -class 35 [ 1280 bytes ] : 42 objs; 0.1 MiB; 15.8 cum MiB -class 36 [ 1408 bytes ] : 29 objs; 0.0 MiB; 15.9 cum MiB -class 37 [ 1536 bytes ] : 30 objs; 0.0 MiB; 15.9 cum MiB -class 38 [ 1792 bytes ] : 213 objs; 0.4 MiB; 16.3 cum MiB -class 39 [ 2048 bytes ] : 523 objs; 1.0 MiB; 17.3 cum MiB -class 40 [ 2304 bytes ] : 14 objs; 0.0 MiB; 17.3 cum MiB -class 41 [ 2560 bytes ] : 7 objs; 0.0 MiB; 17.3 cum MiB -class 45 [ 4096 bytes ] : 31 objs; 0.1 MiB; 17.5 cum MiB -class 46 [ 4608 bytes ] : 5 objs; 0.0 MiB; 17.5 cum MiB -class 47 [ 5120 bytes ] : 1 objs; 0.0 MiB; 17.5 cum MiB -class 49 [ 6656 bytes ] : 4 objs; 0.0 MiB; 17.5 cum MiB -class 50 [ 8192 bytes ] : 36 objs; 0.3 MiB; 17.8 cum MiB -class 51 [ 9216 bytes ] : 38 objs; 0.3 MiB; 18.1 cum MiB -class 55 [ 16384 bytes ] : 11 objs; 0.2 MiB; 18.3 cum MiB -class 56 [ 20480 bytes ] : 4 objs; 0.1 MiB; 18.4 cum MiB -class 58 [ 26624 bytes ] : 2 objs; 0.1 MiB; 18.4 cum MiB -class 59 [ 32768 bytes ] : 9 objs; 0.3 MiB; 18.7 cum MiB -class 60 [ 40960 bytes ] : 4 objs; 0.2 MiB; 18.9 cum MiB -class 63 [ 65536 bytes ] : 9 objs; 0.6 MiB; 19.4 cum MiB -class 71 [ 131072 bytes ] : 9 objs; 1.1 MiB; 20.6 cum MiB -class 72 [ 139264 bytes ] : 4 objs; 0.5 MiB; 21.1 cum MiB -class 87 [ 262144 bytes ] : 9 objs; 2.2 MiB; 23.3 cum MiB ------------------------------------------------- -PageHeap: 3 sizes; 11.2 MiB free; 1.2 MiB unmapped ------------------------------------------------- - 1 pages * 84 spans ~ 0.7 MiB; 0.7 MiB cum; unmapped: 0.7 MiB; 0.7 MiB cum - 14 pages * 1 spans ~ 0.1 MiB; 0.8 MiB cum; unmapped: 0.1 MiB; 0.8 MiB cum - 53 pages * 1 spans ~ 0.4 MiB; 1.2 MiB cum; unmapped: 0.4 MiB; 1.2 MiB cum ->255 large * 1 spans ~ 11.2 MiB; 12.4 MiB cum; unmapped: 0.0 MiB; 1.2 MiB cum - +

    Response

    + ___ Begin jemalloc statistics ___ +Version: 4.5.0-3-g17c897976c60b0e6e4f4a365c751027244dada7a +Assertions disabled +config.malloc_conf: "" +Run-time option settings: + opt.abort: false + opt.lg_chunk: 21 + opt.dss: "secondary" + opt.narenas: 1 + opt.purge: "ratio" + opt.lg_dirty_mult: 3 (arenas.lg_dirty_mult: 3) + opt.junk: "false" + opt.quarantine: 0 + opt.redzone: false + opt.zero: false + opt.tcache: true + opt.lg_tcache_max: 15 + opt.thp: true + opt.prof: true + opt.prof_prefix: "jeprof" + opt.prof_active: false (prof.active: false) + opt.prof_thread_active_init: true (prof.thread_active_init: true) + opt.lg_prof_sample: 19 (prof.lg_sample: 19) + opt.prof_accum: false + opt.lg_prof_interval: -1 + opt.prof_gdump: false + opt.prof_final: false + opt.prof_leak: false + opt.stats_print: false +Arenas: 1 +Min active:dirty page ratio per arena: 8:1 +Quantum size: 16 +Page size: 4096 +Maximum thread-cached size class: 32768 +Allocated: 346524744, active: 347295744, metadata: 3015888, resident: 349773824, mapped: 352321536, retained: 0 +Current active ceiling: 348127232 + +Merged arenas stats: +assigned threads: 26 +dss allocation precedence: N/A +min active:dirty page ratio: N/A +purging: dirty: 55, sweeps: 22, madvises: 103, purged: 734 + allocated nmalloc ndalloc nrequests +small: 54406216 371163 97660 3572656 +large: 2711552 800 731 1113 +huge: 289406976 2 0 2 +total: 346524744 371965 98391 3573771 +active: 347295744 +mapped: 350224384 +retained: 0 +metadata: mapped: 1544192, allocated: 764112 +bins: size ind allocated nmalloc ndalloc nrequests curregs curruns regs pgs util nfills nflushes newruns reruns + 8 0 24920 11111 7996 249538 3115 8 512 1 0.760 268 225 11 42 + 16 1 172144 16287 5528 165964 10759 45 256 1 0.933 854 262 52 73 + 32 2 2718176 94573 9630 1065895 84943 666 128 1 0.996 2687 370 680 912 + 48 3 1189920 31571 6781 602628 24790 99 256 3 0.978 534 264 112 159 + 64 4 1049984 55777 39371 695278 16406 260 64 1 0.985 17213 798 685 456 + 80 5 3011280 40424 2783 123762 37641 149 256 5 0.986 664 180 150 92 + 96 6 460704 9162 4363 50856 4799 40 128 3 0.937 748 145 51 90 + 112 7 2890608 29796 3987 212934 25809 101 256 7 0.998 771 249 113 94 + 128 8 118016 2300 1378 31350 922 34 32 1 0.847 188 190 43 125 + 160 9 1146240 8407 1243 57356 7164 58 128 5 0.964 822 122 59 49 + 192 10 713856 7276 3558 10311 3718 65 64 3 0.893 305 179 68 265 + 224 11 847392 6053 2270 8692 3783 31 128 7 0.953 139 130 35 78 + 256 12 356096 1984 593 54486 1391 89 16 1 0.976 215 163 95 108 + 320 13 1066560 5149 1816 24729 3333 54 64 5 0.964 239 148 61 62 + 384 14 1202304 3680 549 6044 3131 99 32 3 0.988 261 90 104 49 + 448 15 977088 3072 891 6756 2181 35 64 7 0.973 125 95 40 52 + 512 16 6400512 12868 367 28954 12501 1567 8 1 0.997 1871 137 1579 145 + 640 17 11372160 18594 825 83906 17769 559 32 5 0.993 1389 109 566 48 + 768 18 1431552 2058 194 12499 1864 117 16 3 0.995 200 55 126 27 + 896 19 727552 1294 482 3584 812 27 32 7 0.939 100 99 32 55 + 1024 20 389120 761 381 1568 380 95 4 1 1 156 159 152 123 + 1280 21 627200 765 275 1120 490 32 16 5 0.957 92 87 43 56 + 1536 22 3457536 2395 144 6091 2251 283 8 3 0.994 339 70 289 59 + 1792 23 94976 195 142 464 53 4 16 7 0.828 38 68 8 14 + 2048 24 4489216 2522 330 2888 2192 1096 2 1 1 392 139 1210 129 + 2560 25 227840 182 93 179 89 12 8 5 0.927 34 53 18 27 + 3072 26 279552 217 126 232 91 24 4 3 0.947 35 61 42 45 + 3584 27 297472 173 90 216 83 11 8 7 0.943 32 54 17 26 + 4096 28 1335296 1090 764 62772 326 326 1 1 1 244 175 1090 0 + 5120 29 916480 298 119 440 179 45 4 5 0.994 42 59 61 59 + 6144 30 946176 298 144 402 154 79 2 3 0.974 55 74 137 50 + 7168 31 609280 191 106 233 85 23 4 7 0.923 35 61 37 57 + 8192 32 1368064 299 132 301 167 167 1 2 1 114 82 299 0 + 10240 33 839680 161 79 138 82 41 2 5 1 33 57 69 36 + 12288 34 393216 114 82 50 32 32 1 3 1 17 47 114 0 + 14336 35 258048 66 48 40 18 9 2 7 1 9 38 32 4 +large: size ind allocated nmalloc ndalloc nrequests curruns + 16384 36 180224 95 84 134 11 + 20480 37 143360 11 4 46 7 + 24576 38 122880 8 3 48 5 + 28672 39 774144 54 27 86 27 + 32768 40 196608 83 77 250 6 + 40960 41 81920 42 40 42 2 + 49152 42 0 76 76 76 0 + 57344 43 458752 104 96 104 8 + 65536 44 0 306 306 306 0 + 81920 45 0 16 16 16 0 + 98304 46 98304 1 0 1 1 + --- + 131072 48 131072 3 2 3 1 + --- + 524288 56 524288 1 0 1 1 + --- +huge: size ind allocated nmalloc ndalloc nrequests curhchunks + --- + 20971520 77 20971520 1 0 1 1 + --- + 268435456 92 268435456 1 0 1 1 + --- +--- End jemalloc statistics ---
    diff --git a/content/cli/cbstats/cbstats-checkpoint.dita b/content/cli/cbstats/cbstats-checkpoint.dita index 55e336192d..25080f0e62 100644 --- a/content/cli/cbstats/cbstats-checkpoint.dita +++ b/content/cli/cbstats/cbstats-checkpoint.dita @@ -3,21 +3,21 @@ <cmdname>checkpoint</cmdname> - Provides detailed information on per-vbucket checkpoint data structure. + Provides detailed information on per-vBucket checkpoint data structure.
    Syntax

    Request syntax:

    - cbstats [host]:11210 checkpoint + cbstats host:11210 [common options] checkpoint
    Description -

    This command provides detailed information on per-vbucket checkpoint data structure. -

    +

    This command provides detailed information on per-vBucket checkpoint data structure.

    Requesting checkpoint stats impacts the performance and should not be used frequently. -
    +

    The identifier for each vBucket statistic begins with the string vb_ followed by + the vBucket ID and a colon.

    @@ -33,17 +33,28 @@ - cursor_name:cursor_checkpoint_id - Checkpoint ID at which the cursor ‘cursor_name’ is pointing - now. + cursor_name:cursor_checkpoint_id + Checkpoint ID at which the cursor cursor_name is pointing now. + + + cursor_name:cursor_seqno + The seqno at which the cursor cursor_name is pointing now. + + + cursor_name:num_visits + Number of times a batch of items have been drained from a checkpoint of cursor_name. + + + cursor_name:num_items_for_cursor + Number of items remaining for the cursor cursor_name. open_checkpoint_id ID of the current open checkpoint. - num_tap_cursors - Number of referencing TAP cursors. + num_conn_cursors + Number of referencing DCP cursors. num_checkpoint_items @@ -61,14 +72,9 @@ num_items_for_persistence Number of items remaining for persistence. - - checkpoint_extension - True if the open checkpoint is in the extension mode. - state - The state of the vbucket for which this checkpoint contfaiuns - data. + The state of the vBucket for which this checkpoint contains data. last_closed_checkpoint_id @@ -78,71 +84,55 @@ persisted_checkpoint_id The last persisted checkpoint number. + + mem_usage + Total memory taken up by items in all checkpoints under given manager. +
    - +
    Options -

    The following are the command options:

    - - checkpoint options - - - - - - Options - Description - - - - - -h, --help - Shows the help message and exits. - - - -b [bucket-name] - The vBucket to get the status from. Default: default - - - -p [password] - The password for the vBucket if one exists. - - - -
    +

    There are no options for this command. For common cbstats options, see .

    Example -

    Request example:

    - cbstats 10.5.2.117:11210 checkpoint + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample checkpoint

    Response:

    - - vb_0:last_closed_checkpoint_id: 1 - vb_0:num_checkpoint_items: 1 - vb_0:num_checkpoints: 1 - vb_0:num_items_for_persistence: 0 - vb_0:num_open_checkpoint_items: 0 - vb_0:num_tap_cursors: 0 - vb_0:open_checkpoint_id: 2 - vb_0:persisted_checkpoint_id: 1 - vb_0:state: active + vb_0:last_closed_checkpoint_id: 4 + vb_0:mem_usage: 121 + vb_0:num_checkpoint_items: 2 + vb_0:num_checkpoints: 1 + vb_0:num_conn_cursors: 1 + vb_0:num_items_for_persistence: 0 + vb_0:num_open_checkpoint_items: 0 + vb_0:open_checkpoint_id: 5 + vb_0:persisted_checkpoint_id: 4 + vb_0:persistence:cursor_checkpoint_id: 5 + vb_0:persistence:cursor_seqno: 10 + vb_0:persistence:num_visits: 6 + vb_0:state: active ... - vb_9:last_closed_checkpoint_id: 1 - vb_9:num_checkpoint_items: 1 - vb_9:num_checkpoints: 1 - vb_9:num_items_for_persistence: 0 - vb_9:num_open_checkpoint_items: 0 - vb_9:num_tap_cursors: 0 - vb_9:open_checkpoint_id: 2 - vb_9:persisted_checkpoint_id: 1 - vb_9:state: active - + vb_9:last_closed_checkpoint_id: 3 + vb_9:mem_usage: 121 + vb_9:num_checkpoint_items: 2 + vb_9:num_checkpoints: 1 + vb_9:num_conn_cursors: 1 + vb_9:num_items_for_persistence: 0 + vb_9:num_open_checkpoint_items: 0 + vb_9:open_checkpoint_id: 4 + vb_9:persisted_checkpoint_id: 3 + vb_9:persistence:cursor_checkpoint_id: 4 + vb_9:persistence:cursor_seqno: 6 + vb_9:persistence:num_visits: 6 + vb_9:state: active
    diff --git a/content/cli/cbstats/cbstats-dcp-vbtakeover.dita b/content/cli/cbstats/cbstats-dcp-vbtakeover.dita new file mode 100644 index 0000000000..de82f15858 --- /dev/null +++ b/content/cli/cbstats/cbstats-dcp-vbtakeover.dita @@ -0,0 +1,52 @@ + + + + + <cmdname>dcp-vbtakeover</cmdname> + Tracks the progress of rebalance using Database Change Protocol (DCP). + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] dcp-vbtakeover vb name +
    + +
    + Description +

    For internal use only. This command is used by cluster manager (ns_server) to track the progress of rebalance + using DCP.

    +
    + +
    + Options + + dcp-vbtakeover options + + + + + + Option + Description + + + + + vb + vBucket ID. + + + name + Stream name. + + + +
    +

    For common cbstats options, see .

    +
    + +     + +     diff --git a/content/cli/cbstats/cbstats-items.dita b/content/cli/cbstats/cbstats-items.dita new file mode 100644 index 0000000000..23ae5d0e77 --- /dev/null +++ b/content/cli/cbstats/cbstats-items.dita @@ -0,0 +1,45 @@ + + + + + <cmdname>items</cmdname> + Gives statistics about items stored in the bucket. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] items +
    + +
    + Description +

    For memcached buckets only. This command gives statistics about items stored in the specified bucket, or the default bucket if none is specified.

    +
    + +
    + Options +

    There are no options for this command. For common cbstats options, see .

    +
    + +
    + Example + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b orders items + +

    Response

    + items:2:age: 21766 + items:2:evicted: 0 + items:2:evicted_nonzero: 0 + items:2:evicted_time: 0 + items:2:number: 5 + items:2:outofmemory: 0 + items:2:reclaimed: 0 + items:2:tailrepairs: 0 +
    + +     + +     diff --git a/content/cli/cbstats/cbstats-key.dita b/content/cli/cbstats/cbstats-key.dita index 64fe1bb76b..0aa4da19e0 100644 --- a/content/cli/cbstats/cbstats-key.dita +++ b/content/cli/cbstats/cbstats-key.dita @@ -6,7 +6,7 @@
    Syntax

    Request syntax:

    - cbstats [hostname]:11210 key [keyname] [vbid] + cbstats host:11210 [common options] key keyname vbid
    Description

    The following key information is available.

    @@ -25,22 +25,17 @@ key_cas - The keys current cas value. - KV - - - key_data_age - How long the key has waited for its value to be persisted (0 if clean). + The key’s current cas value, as a decimal integer. KV key_exptime - Expiration time from the epoch. + Expiration time from the epoch, as a decimal integer. KV key_flags - Flags for this key. + Flags for this key, as a decimal integer. KV @@ -48,30 +43,59 @@ If the value is not yet persisted. KV + + key_is_resident + If the value is resident in memory. + KV + key_valid - See description below*. + See description below. * V key_vb_state - The vbucket state of this key. + The vBucket state of this key. KV -

    *key_valid= can have the following responses:

    -
      -
    • this_is_a_bug: Some case we didn’t take care of.
      • -
    • dirty:The value in memory has not been persisted yet.
      • -
    • length_mismatch: The key length in memory doesn’t match the length on the disk.
      • -
    • data_mismatch: The data in memory doesn’t match the data on disk.
      • -
    • flags_mismatch: The flags in memory don’t match the flags on disk.
      • -
    • valid: The key is both on disk and in memory
      • -
    • ram_but_not_disk: The value doesn’t exist yet on disk.
      • -
    • item_deleted: The item has been deleted.
      • -
    +

    * key_valid can have the following responses:

    +
    + +
    this_is_a_bug
    +
    Some case we didn’t take care of.
    +     + +
    dirty
    +
    The value in memory has not been persisted yet.
    +     + +
    length_mismatch
    +
    The key length in memory doesn’t match the length on the disk.
    +     + +
    data_mismatch
    +
    The data in memory doesn’t match the data on disk.
    +     + +
    flags_mismatch
    +
    The flags in memory don’t match the flags on disk.
    +     + +
    valid
    +
    The key is both on disk and in memory.
    +     + +
    ram_but_not_disk
    +
    The value doesn’t exist yet on disk.
    +     + +
    item_deleted
    +
    The item has been deleted.
    +     +
    @@ -88,27 +112,58 @@ - KEYNAME + keyname Name of the key. - VBID - vBucket ID. + vbid + vBucket ID. † + + +

    † To find the vBucket ID associated with a given key, you can use the cbc-hash + command, which is available as part of the Couchbase C SDK (libcouchbase).

    +
      +
    • For more information on the Couchbase C SDK, see .
      • +
    • For more information on the cbc-hash command, see + Couchbase C Client: + cbc(1) - Couchbase Client Commandline Utility.
      • +
    +     + +

    For common cbstats options, see .

    -
    Example + +
    + Example 1: Find the vBucket ID associated with a given key + +

    Request

    + cbc-hash 21st_amendment_brewery_cafe -U couchbase://localhost/beer-sample -u Administrator -P password + +

    Response

    + 21st_amendment_brewery_cafe: [vBucket=553, Index=1] Server: 10.142.180.102:11210, CouchAPI: http://10.142.180.102:8092/beer-sample +Replica #0: Index=0, Host=10.142.180.101:11210 +
    + +
    + Example 2: Get key information +

    Request

    - cbstats 10.5.2.54:11210 key foo 115 -

    Response

    -

    Example response for the key foo:

    - key_cas: 41206791218746 - key_exptime: 0 - key_flags: 0 - key_is_dirty: 0 + cbstats localhost:11210 -u Administrator -p password -b beer-sample key 21st_amendment_brewery_cafe 553 + +

    Response

    + stats for key 21st_amendment_brewery_cafe + key_cas: 1532081261254082560 + key_exptime: 0 + key_flags: 2 + key_is_dirty: false + key_is_resident: true + key_vb_state: replica
    + diff --git a/content/cli/cbstats/cbstats-prev-vbucket.dita b/content/cli/cbstats/cbstats-prev-vbucket.dita new file mode 100644 index 0000000000..6cb2127471 --- /dev/null +++ b/content/cli/cbstats/cbstats-prev-vbucket.dita @@ -0,0 +1,176 @@ + + + + + <cmdname>prev-vbucket</cmdname> + Lists the previous state of each vBucket. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] prev-vbucket +
    + +
    + Description +

    This command lists the previous state of each vBucket.

    +
    + +
    + Options +

    There are no options for this command. For common cbstats options, see .

    +
    + +
    + Example + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample prev-vbucket + +

    Response

    +

    The output is quite lengthy, so this sample is truncated.

    + vb_0: active + vb_1: active + vb_10: active + vb_100: active + vb_1000: active + vb_1001: active + vb_1002: active + vb_1003: active + vb_1004: active + vb_1005: active + vb_1006: active + vb_1007: active + vb_1008: active + vb_1009: active + vb_101: active + vb_1010: active + vb_1011: active + vb_1012: active + vb_1013: active + vb_1014: active + vb_1015: active + vb_1016: active + vb_1017: active + vb_1018: active + vb_1019: active + vb_102: active + vb_1020: active + vb_1021: active + vb_1022: active + vb_1023: active + vb_103: active + vb_104: active + vb_105: active + vb_106: active + vb_107: active + vb_108: active + vb_109: active + vb_11: active + vb_110: active + vb_111: active + vb_112: active + vb_113: active + vb_114: active + vb_115: active + vb_116: active + vb_117: active + vb_118: active + vb_119: active + vb_12: active + vb_120: active + vb_121: active + vb_122: active + vb_123: active + vb_124: active + vb_125: active + vb_126: active + vb_127: active + vb_128: active + vb_129: active + vb_13: active + vb_130: active + vb_131: active + vb_132: active + vb_133: active + vb_134: active + vb_135: active + vb_136: active + vb_137: active + vb_138: active + vb_139: active + vb_14: active + vb_140: active + vb_141: active + vb_142: active + vb_143: active + vb_144: active + vb_145: active + vb_146: active + vb_147: active + vb_148: active + vb_149: active + vb_15: active + vb_150: active + vb_151: active + vb_152: active + vb_153: active + vb_154: active + vb_155: active + vb_156: active + vb_157: active + vb_158: active + vb_159: active + vb_16: active + vb_160: active + vb_161: active + vb_162: active + vb_163: active + vb_164: active + vb_165: active + vb_166: active + vb_167: active + vb_168: active + vb_169: active + vb_17: active + vb_170: active + vb_171: active + vb_172: active + vb_173: active + vb_174: active + vb_175: active + vb_176: active + vb_177: active + vb_178: active + vb_179: active + vb_18: active + vb_180: active + vb_181: active + vb_182: active + vb_183: active + vb_184: active + vb_185: active + vb_186: active + vb_187: active + vb_188: active + vb_189: active + vb_19: active + vb_190: active + vb_191: active + vb_192: active + vb_193: active + vb_194: active + vb_195: active + vb_196: active + vb_197: active + vb_198: active + vb_199: active + ... + vb_999: active +
    + +     + +     diff --git a/content/cli/cbstats/cbstats-raw.dita b/content/cli/cbstats/cbstats-raw.dita new file mode 100644 index 0000000000..05bd55f227 --- /dev/null +++ b/content/cli/cbstats/cbstats-raw.dita @@ -0,0 +1,155 @@ + + + + + <cmdname>raw</cmdname> + Issues a request for specific statistics. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] raw argument +
    + +
    + Description +

    For internal use only. This command issues a request for specific statistics in raw format.

    +
    + +
    + Options + + raw options + + + + + + Option + Description + + + + + argument + The subcommand to pass through to the raw command. + + + +
    +

    For common cbstats options, see .

    +
    + +
    + Example + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample raw timings + +

    Response

    +

    The output is quite lengthy, so this sample is truncated.

    + access_scanner_0,1: 0 + access_scanner_1,2: 0 + access_scanner_1024,2048: 0 + access_scanner_1048576,2097152: 0 + access_scanner_1073741824,18446744073709551615: 0 + access_scanner_128,256: 0 + access_scanner_131072,262144: 0 + access_scanner_134217728,268435456: 0 + access_scanner_16,32: 0 + access_scanner_16384,32768: 0 + access_scanner_16777216,33554432: 0 + access_scanner_2,4: 0 + access_scanner_2048,4096: 0 + access_scanner_2097152,4194304: 0 + access_scanner_256,512: 0 + access_scanner_262144,524288: 0 + access_scanner_268435456,536870912: 0 + access_scanner_32,64: 0 + access_scanner_32768,65536: 0 + access_scanner_33554432,67108864: 0 + access_scanner_4,8: 0 + access_scanner_4096,8192: 0 + access_scanner_4194304,8388608: 0 + access_scanner_512,1024: 0 + access_scanner_524288,1048576: 0 + access_scanner_536870912,1073741824: 0 + access_scanner_64,128: 0 + access_scanner_65536,131072: 0 + access_scanner_67108864,134217728: 0 + access_scanner_8,16: 0 + access_scanner_8192,16384: 0 + access_scanner_8388608,16777216: 0 + ... + checkpoint_remover_0,1: 0 + checkpoint_remover_1,2: 0 + checkpoint_remover_1024,2048: 1 + checkpoint_remover_1048576,2097152: 0 + checkpoint_remover_1073741824,18446744073709551615: 0 + checkpoint_remover_128,256: 0 + checkpoint_remover_131072,262144: 0 + checkpoint_remover_134217728,268435456: 0 + checkpoint_remover_16,32: 0 + checkpoint_remover_16384,32768: 9 + checkpoint_remover_16777216,33554432: 0 + checkpoint_remover_2,4: 0 + checkpoint_remover_2048,4096: 4506 + checkpoint_remover_2097152,4194304: 0 + checkpoint_remover_256,512: 0 + checkpoint_remover_262144,524288: 0 + checkpoint_remover_268435456,536870912: 0 + checkpoint_remover_32,64: 0 + checkpoint_remover_32768,65536: 2 + checkpoint_remover_33554432,67108864: 0 + checkpoint_remover_4,8: 0 + checkpoint_remover_4096,8192: 770 + checkpoint_remover_4194304,8388608: 0 + checkpoint_remover_512,1024: 0 + checkpoint_remover_524288,1048576: 0 + checkpoint_remover_536870912,1073741824: 0 + checkpoint_remover_64,128: 0 + checkpoint_remover_65536,131072: 0 + checkpoint_remover_67108864,134217728: 0 + checkpoint_remover_8,16: 0 + checkpoint_remover_8192,16384: 101 + checkpoint_remover_8388608,16777216: 0 + ... + store_cmd_0,1: 0 + store_cmd_1,2: 0 + store_cmd_1024,2048: 0 + store_cmd_1048576,2097152: 0 + store_cmd_1073741824,18446744073709551615: 0 + store_cmd_128,256: 0 + store_cmd_131072,262144: 0 + store_cmd_134217728,268435456: 0 + store_cmd_16,32: 0 + store_cmd_16384,32768: 0 + store_cmd_16777216,33554432: 0 + store_cmd_2,4: 0 + store_cmd_2048,4096: 0 + store_cmd_2097152,4194304: 0 + store_cmd_256,512: 0 + store_cmd_262144,524288: 0 + store_cmd_268435456,536870912: 0 + store_cmd_32,64: 0 + store_cmd_32768,65536: 0 + store_cmd_33554432,67108864: 0 + store_cmd_4,8: 0 + store_cmd_4096,8192: 0 + store_cmd_4194304,8388608: 0 + store_cmd_512,1024: 0 + store_cmd_524288,1048576: 0 + store_cmd_536870912,1073741824: 0 + store_cmd_64,128: 0 + store_cmd_65536,131072: 0 + store_cmd_67108864,134217728: 0 + store_cmd_8,16: 0 + store_cmd_8192,16384: 0 + store_cmd_8388608,16777216: 0 +
    + +     + +     diff --git a/content/cli/cbstats/cbstats-responses.dita b/content/cli/cbstats/cbstats-responses.dita new file mode 100644 index 0000000000..39bb20a8b9 --- /dev/null +++ b/content/cli/cbstats/cbstats-responses.dita @@ -0,0 +1,114 @@ + + + + + <cmdname>responses</cmdname> + Summarizes the response codes a bucket has returned. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] responses [all] +
    + +
    + Description +

    This command summarizes the response codes that the specified bucket has returned, or the default bucket if none is specified.

    +
    + +
    + Options + + responses options + + + + + + Option + Description + + + + + all + If this argument is omitted, the command summarizes only the responses that the bucket has returned. If this argument is specified, the command summarizes all possible responses, including those the bucket has not returned. + + + +
    +

    For common cbstats options, see .

    +
    + +
    + Example 1 + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample responses + +

    Response

    + EINVAL: 2 + KEY_ENOENT: 28 + SUCCESS: 180932 +
    + +
    + Example 2 + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample responses all + +

    Response

    + AUTH_CONTINUE: 0 + AUTH_ERROR: 0 + AUTH_STALE: 0 + DELTA_BADVAL: 0 + E2BIG: 0 + EACCESS: 0 + EBUSY: 0 + EINTERNAL: 0 + EINVAL: 2 + ENOMEM: 0 + ERANGE: 0 + ETMPFAIL: 0 + KEY_EEXISTS: 0 + KEY_ENOENT: 28 + LOCKED: 0 + NOT_INITIALIZED: 0 + NOT_MY_VBUCKET: 0 + NOT_STORED: 0 + NOT_SUPPORTED: 0 + NO_BUCKET: 0 + ROLLBACK: 0 + SUBDOC_DELTA_EINVAL: 0 + SUBDOC_DOC_NOTJSON: 0 + SUBDOC_INVALID_COMBO: 0 + SUBDOC_INVALID_XATTR_ORDER: 0 + SUBDOC_MULTI_PATH_FAILURE: 0 + SUBDOC_MULTI_PATH_FAILURE_DELETED: 0 + SUBDOC_NUM_ERANGE: 0 + SUBDOC_PATH_E2BIG: 0 + SUBDOC_PATH_E2DEEP: 0 + SUBDOC_PATH_EEXISTS: 0 + SUBDOC_PATH_EINVAL: 0 + SUBDOC_PATH_ENOENT: 0 + SUBDOC_PATH_MISMATCH: 0 + SUBDOC_SUCCESS_DELETED: 0 + SUBDOC_VALUE_CANTINSERT: 0 + SUBDOC_VALUE_ETOODEEP: 0 + SUBDOC_XATTR_CANT_MODIFY_VATTR: 0 + SUBDOC_XATTR_INVALID_FLAG_COMBO: 0 + SUBDOC_XATTR_INVALID_KEY_COMBO: 0 + SUBDOC_XATTR_UNKNOWN_MACRO: 0 + SUBDOC_XATTR_UNKNOWN_VATTR: 0 + SUCCESS: 180932 + UNKNOWN_COLLECTION: 0 + UNKNOWN_COMMAND: 0 + XATTR_EINVAL: 0 +
    + +     + +     diff --git a/content/cli/cbstats/cbstats-slabs.dita b/content/cli/cbstats/cbstats-slabs.dita new file mode 100644 index 0000000000..524cfcb24e --- /dev/null +++ b/content/cli/cbstats/cbstats-slabs.dita @@ -0,0 +1,46 @@ + + + + + <cmdname>slabs</cmdname> + Gives statistics on current memory allocation. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] slabs +
    + +
    + Description +

    For memcached buckets only. This command gives statistics on current memory allocation (slab allocator).

    +
    + +
    + Options +

    There are no options for this command. For common cbstats options, see .

    +
    + +
    + Example + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b orders slabs + +

    Response

    + 2:chunk_size: 144 + 2:chunks_per_page: 7281 + 2:free_chunks: 0 + 2:free_chunks_end: 7276 + 2:mem_requested: 661 + 2:total_chunks: 7281 + 2:total_pages: 1 + 2:used_chunks: 5 + active_slabs: 1 + total_malloced: 1048464 +
    +     + +     diff --git a/content/cli/cbstats/cbstats-tap-vbtakeover.dita b/content/cli/cbstats/cbstats-tap-vbtakeover.dita new file mode 100644 index 0000000000..16e31e8969 --- /dev/null +++ b/content/cli/cbstats/cbstats-tap-vbtakeover.dita @@ -0,0 +1,55 @@ + + + + + <cmdname>tap-vbtakeover</cmdname> + Tracks the progress of rebalance using TAP. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] tap-vbtakeover vb name +
    + +
    + Description +

    For internal use only. This command is used by cluster manager (ns_server) to track the progress of rebalance + using TAP.

    +

    TAP is an internal protocol that streams information about data changes between cluster + nodes. It was replaced by DCP and removed from Couchbase Server in version 5.0. This command enables you to + retrieve statistics from clusters running legacy versions of Couchbase Server.

    +
    + +
    + Options + + tap-vbtakeover options + + + + + + Option + Description + + + + + vb + vBucket ID. + + + name + Stream name. + + + +
    +

    For common cbstats options, see .

    +
    + +     + +     diff --git a/content/cli/cbstats/cbstats-tasks.dita b/content/cli/cbstats/cbstats-tasks.dita new file mode 100644 index 0000000000..c77a0fab91 --- /dev/null +++ b/content/cli/cbstats/cbstats-tasks.dita @@ -0,0 +1,110 @@ + + + + + <cmdname>tasks</cmdname> + Gives details on any running background tasks. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] tasks [sort column] +
    + +
    + Description +

    This command gives details on any running background tasks. The first three lines of output contain a summary of the tasks. The rest of the output lists the task details in a table.

    +
    + +
    + Options + + tasks options + + + + + + Option + Description + + + + + sort column + The column by which to sort the details. + + + +
    +

    For common cbstats options, see .

    +
    + +
    + Example + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample tasks Name + +

    Response

    + Tasks Writer Reader AuxIO NonIO Total Uptime: 7:42:56 +Running 0 0 0 0 0 +All 10 8 10 20 48 + +TID Pri St Bucket SleepFor Runtime TotalRun #Runs Type Name Addr Descr. + + 75 3 S travel-sample 9:55:29.28 0:00.00 0:00.00 0 AuxIO AccessScanner 0x00007f4718e3ce50 Generating access log + 79 3 S beer-sample 9:55:29.45 0:00.00 0:00.00 0 AuxIO AccessScanner 0x00007f47191b3f50 Generating access log + 81 5 S beer-sample inf 0:00.00 0:00.00 0 AuxIO ActiveStreamCheckpointProcessorTask 0x00007f472411bd90 Process checkpoint(s) for DCP producer eq_dcpq:secidx:proj-beer-sample-MAINT_STREAM_TOPIC_8d862a0c72cf81a62cb2d4d582e295dc-16965435698462730570/0 + 82 5 S beer-sample inf 0:00.00 0:00.00 0 AuxIO ActiveStreamCheckpointProcessorTask 0x00007f4744144790 Process checkpoint(s) for DCP producer eq_dcpq:secidx:proj-beer-sample-MAINT_STREAM_TOPIC_8d862a0c72cf81a62cb2d4d582e295dc-16965435698462730570/1 + 83 5 S beer-sample inf 0:00.00 0:00.00 0 AuxIO ActiveStreamCheckpointProcessorTask 0x00007f471ad8ad90 Process checkpoint(s) for DCP producer eq_dcpq:secidx:proj-beer-sample-MAINT_STREAM_TOPIC_8d862a0c72cf81a62cb2d4d582e295dc-16965435698462730570/2 + 84 5 S beer-sample inf 0:00.00 0:00.00 0 AuxIO ActiveStreamCheckpointProcessorTask 0x00007f4744158390 Process checkpoint(s) for DCP producer eq_dcpq:secidx:proj-beer-sample-MAINT_STREAM_TOPIC_8d862a0c72cf81a62cb2d4d582e295dc-16965435698462730570/3 + 85 5 S travel-sample inf 0:00.00 0:00.00 0 AuxIO ActiveStreamCheckpointProcessorTask 0x00007f4744144910 Process checkpoint(s) for DCP producer eq_dcpq:secidx:proj-travel-sample-MAINT_STREAM_TOPIC_8d862a0c72cf81a62cb2d4d582e295dc-5207537980771054579/0 + 86 5 S travel-sample inf 0:00.00 0:00.00 0 AuxIO ActiveStreamCheckpointProcessorTask 0x00007f471ad8af10 Process checkpoint(s) for DCP producer eq_dcpq:secidx:proj-travel-sample-MAINT_STREAM_TOPIC_8d862a0c72cf81a62cb2d4d582e295dc-5207537980771054579/1 + 87 5 S travel-sample inf 0:00.00 0:00.00 0 AuxIO ActiveStreamCheckpointProcessorTask 0x00007f4744158510 Process checkpoint(s) for DCP producer eq_dcpq:secidx:proj-travel-sample-MAINT_STREAM_TOPIC_8d862a0c72cf81a62cb2d4d582e295dc-5207537980771054579/2 + 88 5 S travel-sample inf 0:00.00 0:00.00 0 AuxIO ActiveStreamCheckpointProcessorTask 0x00007f472411bf10 Process checkpoint(s) for DCP producer eq_dcpq:secidx:proj-travel-sample-MAINT_STREAM_TOPIC_8d862a0c72cf81a62cb2d4d582e295dc-5207537980771054579/3 + 5 6 S travel-sample 0:00.43 0:00.00 0:01.87 5553 NonIO ClosedUnrefCheckpointRemoverTask 0x00007f474417b090 Removing closed unreferenced checkpoints from memory + 36 6 S beer-sample 0:00.28 0:00.00 0:01.83 5553 NonIO ClosedUnrefCheckpointRemoverTask 0x00007f4725779590 Removing closed unreferenced checkpoints from memory + 19 8 S travel-sample 0:00.71 0:00.00 0:18.02 27740 NonIO ConnManager 0x00007f474417b390 Connection Manager + 50 8 S beer-sample 0:00.27 0:00.00 0:16.99 27740 NonIO ConnManager 0x00007f474417b710 Connection Manager + 18 5 S travel-sample 0:00.01 0:00.00 0:01.53 32503 NonIO ConnNotifierCallback 0x00007f474417b310 DCP connection notifier + 49 5 S beer-sample 0:00.94 0:00.00 0:01.56 32503 NonIO ConnNotifierCallback 0x00007f474417b690 DCP connection notifier + 7 7 S travel-sample 0:06.04 0:00.00 0:19.08 2777 NonIO DefragmenterTask 0x00007f474417b110 Memory defragmenter + 38 7 S beer-sample 0:08.93 0:00.00 0:11.83 2777 NonIO DefragmenterTask 0x00007f474417b590 Memory defragmenter + 3 1 S travel-sample 17:05.26 0:00.00 0:00.00 7 NonIO ExpiredItemPager 0x00007f475fa79e70 Paging expired items. + 34 1 S beer-sample 17:05.32 0:00.00 0:00.00 7 NonIO ExpiredItemPager 0x00007f475fa79fb0 Paging expired items. + 14 5 S travel-sample 0:04.00 0:00.00 0:00.15 13180 Writer FlusherTask 0x00007f475fa7a050 Running a flusher loop: shard 0 + 15 5 S travel-sample 0:04.00 0:00.00 0:00.15 13180 Writer FlusherTask 0x00007f475fa7a0f0 Running a flusher loop: shard 1 + 16 5 S travel-sample 0:04.00 0:00.00 0:00.08 13180 Writer FlusherTask 0x00007f475fa7a190 Running a flusher loop: shard 2 + 17 5 S travel-sample 0:04.00 0:00.00 0:00.04 13180 Writer FlusherTask 0x00007f475fa7a230 Running a flusher loop: shard 3 + 45 5 S beer-sample 0:04.19 0:00.00 0:00.14 13180 Writer FlusherTask 0x00007f471ad4b890 Running a flusher loop: shard 0 + 46 5 S beer-sample 0:04.24 0:00.00 0:00.17 13180 Writer FlusherTask 0x00007f471ad4b930 Running a flusher loop: shard 1 + 47 5 S beer-sample 0:04.19 0:00.00 0:00.08 13180 Writer FlusherTask 0x00007f471ad4b9d0 Running a flusher loop: shard 2 + 48 5 S beer-sample 0:04.24 0:00.00 0:00.10 13180 Writer FlusherTask 0x00007f471ad4ba70 Running a flusher loop: shard 3 + 4 211 S travel-sample 0:00.66 0:00.00 0:08.69 27740 NonIO HashtableResizerTask 0x00007f472569b680 Adjusting hash table sizes. + 35 211 S beer-sample 0:00.08 0:00.00 0:09.01 27740 NonIO HashtableResizerTask 0x00007f471b54b310 Adjusting hash table sizes. + 8 7 S travel-sample 0:00.07 0:00.00 0:06.73 110800 NonIO ItemCompressorTask 0x00007f474417b190 Item Compressor + 39 7 S beer-sample 0:00.01 0:00.00 0:04.05 110800 NonIO ItemCompressorTask 0x00007f474417b610 Item Compressor + 9 7 S travel-sample inf 0:00.00 0:00.00 1 NonIO ItemFreqDecayerTask 0x00007f472569b760 Item frequency count decayer task + 40 7 S beer-sample inf 0:00.00 0:00.00 1 NonIO ItemFreqDecayerTask 0x00007f471b54b3f0 Item frequency count decayer task + 1 1 S travel-sample 0:03.86 0:00.00 0:00.20 5553 NonIO ItemPager 0x00007f475fa79dd0 Paging out items. + 24 1 S beer-sample 0:03.86 0:00.00 0:00.15 5553 NonIO ItemPager 0x00007f475fa79f10 Paging out items. + 10 0 S travel-sample inf 0:00.00 0:00.00 0 Reader MultiBGFetcherTask 0x00007f472569b7d0 Batching background fetch + 11 0 S travel-sample inf 0:00.00 0:00.00 0 Reader MultiBGFetcherTask 0x00007f472569b840 Batching background fetch + 12 0 S travel-sample inf 0:00.00 0:00.00 0 Reader MultiBGFetcherTask 0x00007f472569b8b0 Batching background fetch + 13 0 S travel-sample inf 0:00.00 0:00.00 0 Reader MultiBGFetcherTask 0x00007f472569b920 Batching background fetch + 41 0 S beer-sample inf 0:00.00 0:00.00 0 Reader MultiBGFetcherTask 0x00007f471b54b460 Batching background fetch + 42 0 S beer-sample inf 0:00.00 0:00.00 0 Reader MultiBGFetcherTask 0x00007f471b54b4d0 Batching background fetch + 43 0 S beer-sample inf 0:00.00 0:00.00 0 Reader MultiBGFetcherTask 0x00007f471b54b540 Batching background fetch + 44 0 S beer-sample inf 0:00.00 0:00.00 0 Reader MultiBGFetcherTask 0x00007f471b54b5b0 Batching background fetch + 77 9 S travel-sample 0:16.72 0:00.13 01:05.89 462 Writer StatSnap 0x00007f4718ed6c90 Updating stat snapshot on disk + 80 9 S beer-sample 0:16.72 0:00.12 01:05.66 462 Writer StatSnap 0x00007f4718f6bc70 Updating stat snapshot on disk + 6 10 S travel-sample 0:03.86 0:00.00 0:00.08 5554 NonIO WorkLoadMonitor 0x00007f472569b6f0 Monitoring a workload pattern + 37 10 S beer-sample 0:03.86 0:00.00 0:00.05 5554 NonIO WorkLoadMonitor 0x00007f471b54b380 Monitoring a workload pattern +
    + +     + +     diff --git a/content/cli/cbstats/cbstats-timing.dita b/content/cli/cbstats/cbstats-timing.dita index f589988e53..164c07fb2c 100644 --- a/content/cli/cbstats/cbstats-timing.dita +++ b/content/cli/cbstats/cbstats-timing.dita @@ -6,77 +6,19 @@ Provides timing statistics.
    Syntax -

    The request syntax:

    - cbstats [host]:[dataport] timings +

    Request syntax:

    + cbstats host:11210 [common options] timings
    Description

    The timing stats provide histogram data from high-resolution timers over various operations within the system. This only measures the time spent in the front-end thread for each operation, meaning that the timings may not be - representative if items have to be fetched from disk as part of the operation.

    To - retrieve more accurate timing statistics, use mctimings - instead of timings.
    - -
    Options -

    None

    - -
    - -
    Example - -

    The following example, uses the 10.5.2.117 host and the default port, 11210:

    - cbstats 10.5.2.117:11210 timings - - -

    Response

    - -

    The following is sample output from cbstats timings:

    - -disk_commit (1024 total) - 0 - 1s : (100.00%) 1024 ################################################### - Avg : ( 1s) - get_stats_cmd (30663276 total) - 0 - 1us : ( 0.05%) 14827 - 1us - 2us : ( 6.56%) 1995778 ## - 2us - 4us : ( 41.79%) 10804626 ############## - 4us - 8us : ( 45.20%) 1044043 # - 8us - 16us : ( 45.49%) 89929 - 16us - 32us : ( 45.90%) 124472 - 32us - 64us : ( 46.38%) 148935 - 64us - 128us : ( 56.17%) 2999690 ### - 128us - 256us : ( 68.57%) 3804009 #### - 256us - 512us : ( 69.91%) 411281 - 512us - 1ms : ( 78.77%) 2717402 ### - 1ms - 2ms : ( 96.36%) 5391526 ####### - 2ms - 4ms : ( 99.05%) 826345 # - 4ms - 8ms : ( 99.96%) 278727 - 8ms - 16ms : (100.00%) 11443 - 16ms - 32ms : (100.00%) 217 - 32ms - 65ms : (100.00%) 19 - 65ms - 131ms : (100.00%) 7 - Avg : ( 347us) - disk_vbstate_snapshot (93280 total) - 32us - 64us : ( 15.34%) 14308 ###### - 64us - 128us : ( 74.74%) 55413 ######################### - 128us - 256us : ( 91.39%) 15532 ####### - 256us - 512us : ( 95.69%) 4007 # - 512us - 1ms : ( 99.49%) 3546 # - 1ms - 2ms : ( 99.95%) 423 - 2ms - 4ms : ( 99.99%) 43 - 4ms - 8ms : (100.00%) 4 - 2s - 4s : (100.00%) 4 - Avg : ( 190us) - notify_io (4 total) - 4us - 8us : ( 25.00%) 1 ############ - 16us - 32us : ( 75.00%) 2 ######################## - 32us - 64us : (100.00%) 1 ############ - Avg : ( 17us) - - - -

    The following are the possible return values, which depend on what occurred on the data + representative if items have to be fetched from disk as part of the operation.

    +

    To retrieve more accurate timing statistics, use mctimings + instead of timings.

    +

    The following are the possible return values, which depend on what occurred on the data bucket:

    - Return values - +
    Return values + @@ -88,55 +30,103 @@ disk_commit (1024 total) bg_load - Background fetches waiting for disk + Background fetches waiting for disk. bg_wait - Background fetches waiting in the dispatcher queue + Background fetches waiting in the dispatcher queue. + + + set_with_meta + set_with_meta latencies. + + + access_scanner + Access scanner run times. + + + checkpoint_remover + Checkpoint remover run times. + + + item_pager + Item pager run times. + + + expiry_pager + Expiry pager run times. + + + pending_ops + Client connections blocked for operations in pending vBuckets. data_age - Age of data written to disk + Age of data written to disk. disk_commit - Time waiting for a commit after a batch of updates + Time waiting for a commit after a batch of updates. disk_del - Wait for disk to delete an item + Wait for disk to delete an item. - disk_insert - Wait for disk to store a new item + disk_vb_del + Waiting for disk to delete a vBucket. - disk_vbstate_snapshot - Time spent persisting vbucket state changes + disk_insert + Wait for disk to store a new item. disk_update - Wait time for disk to modify an existing item + Wait time for disk to modify an existing item. get_cmd - Servicing get requests + Servicing get requests. + + + store_cmd + Servicing store requests. + + + arith_cmd + Servicing incr/decr requests. get_stats_cmd - Servicing get_stats requests + Servicing get_stats requests. + + + get_vb_cmd + Servicing vBucket status requests. set_vb_cmd - Servicing vbucket set state commands + Servicing vBucket set state commands. + + + del_vb_cmd + Servicing vBucket deletion commands. + + + chk_persistence_cmd + Waiting for checkpoint persistence. item_alloc_sizes - Item allocation size counters (in bytes) + Item allocation size counters (in bytes). notify_io - Time for waking blocked connections + Time for waking blocked connections. + + + paged_out_time + Time (in seconds) objects are non-resident. storage_age @@ -144,13 +134,107 @@ disk_commit (1024 total) storage. - tap_mutation - Time spent servicing tap mutations + bg_batch_size + Batch size for background fetches. + + + persistence_cursor_get_all_items + Time spent in fetching all items by persistence cursor from checkpoint queues. + + + dcp_cursors_get_all_items + Time spent in fetching all items by all dcp cursors from checkpoint queues. -
    + + The most useful stats for understanding get and set timings are get_cmd and + store_cmd. +
    +
    Options +

    There are no options for this command. For common cbstats options, see + .

    +
    + +
    Example + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample timings + +

    Response

    + checkpoint_remover (101 total) + 2ms - 4ms : ( 12.87%) 13 █████▌ + 4ms - 8ms : ( 94.06%) 82 ██████████████████████████████████▉ + 8ms - 16ms : ( 98.02%) 4 █▋ + 16ms - 32ms : (100.00%) 2 ▊ + Avg : ( 4ms) + dcp_cursors_get_all_items (2 total) + 1us - 2us : ( 50.00%) 1 ██████████████████████ + 2us - 4us : ( 50.00%) 0 + 4us - 8us : (100.00%) 1 ██████████████████████ + Avg : ( 2us) + disk_commit (1 total) + 2ms - 4ms : (100.00%) 1 ████████████████████████████████████████████ + Avg : ( 2ms) + disk_update (1 total) + 4us - 8us : (100.00%) 1 ████████████████████████████████████████████ + Avg : ( 4us) + get_cmd (3 total) + 4m:28s - 8m:56s : (100.00%) 3 ████████████████████████████████████████████ + Avg : ( 4m:28s) + get_stats_cmd (2220 total) + 16s - 33s : ( 0.50%) 11 ▏ + 33s - 1m:07s : ( 4.50%) 89 █▋ + 1m:07s - 2m:14s : ( 14.05%) 212 ███▉ + 2m:14s - 4m:28s : ( 42.12%) 623 ███████████▌ + 4m:28s - 8m:56s : (100.00%) 1285 ███████████████████████▋ + Avg : ( 3m:20s) + item_alloc_sizes (1 total) + 512 - 1KB : (100.00%) 1 ███████████████████████████████████████████████ + Avg : ( 512) + notify_io (4839 total) + 0 - 1us : ( 17.42%) 843 ███████▏ + 1us - 2us : ( 26.04%) 417 ███▌ + 2us - 4us : ( 42.10%) 777 ██████▌ + 4us - 8us : ( 70.04%) 1352 ███████████▍ + 8us - 16us : ( 78.43%) 406 ███▍ + 16us - 32us : ( 90.80%) 599 █████ + 32us - 64us : ( 95.60%) 232 █▉ + 64us - 128us : ( 96.42%) 40 ▎ + 128us - 256us : ( 97.56%) 55 ▍ + 256us - 512us : ( 98.74%) 57 ▍ + 512us - 1ms : ( 99.03%) 14 + 1ms - 2ms : ( 99.24%) 10 + 2ms - 4ms : ( 99.48%) 12 + 4ms - 8ms : ( 99.90%) 20 ▏ + 8ms - 16ms : ( 99.98%) 4 + 16ms - 32ms : (100.00%) 1 + Avg : ( 46us) + persistence_cursor_get_all_items (60416 total) + 0 - 1us : ( 96.83%) 58501 ██████████████████████████████████████▋ + 1us - 2us : ( 99.33%) 1511 █ + 2us - 4us : ( 99.79%) 276 ▏ + 4us - 8us : ( 99.82%) 18 + 8us - 16us : ( 99.83%) 8 + 16us - 32us : ( 99.85%) 11 + 32us - 64us : ( 99.90%) 32 + 64us - 128us : ( 99.94%) 24 + 128us - 256us : ( 99.95%) 5 + 256us - 512us : ( 99.96%) 4 + 512us - 1ms : ( 99.96%) 1 + 1ms - 2ms : ( 99.97%) 4 + 2ms - 4ms : ( 99.98%) 7 + 4ms - 8ms : ( 99.99%) 7 + 8ms - 16ms : (100.00%) 6 + 16ms - 32ms : (100.00%) 1 + Avg : ( 1us) + storage_age (1 total) + 0 - 1us : (100.00%) 1 ████████████████████████████████████████████ + Avg : ( 0) + store_cmd (1 total) + 4m:28s - 8m:56s : (100.00%) 1 ████████████████████████████████████████████ + Avg : ( 4m:28s)
    diff --git a/content/cli/cbstats/cbstats-uuid.dita b/content/cli/cbstats/cbstats-uuid.dita index 1dc81962b2..801d3d9e84 100644 --- a/content/cli/cbstats/cbstats-uuid.dita +++ b/content/cli/cbstats/cbstats-uuid.dita @@ -20,7 +20,7 @@ - + diff --git a/content/cli/cbstats/cbstats-vbucket-details.dita b/content/cli/cbstats/cbstats-vbucket-details.dita new file mode 100644 index 0000000000..cd141ecf4c --- /dev/null +++ b/content/cli/cbstats/cbstats-vbucket-details.dita @@ -0,0 +1,349 @@ + + + + + <cmdname>vbucket-details</cmdname> + Provides details for vBuckets. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] vbucket-details [vbid] +
    + +
    + Description +

    This command provides details for the specified vBucket, or for each vBucket if none is specified.

    + +

    The identifier for each vBucket statistic begins with the string vb_ followed by + the vBucket ID and a colon. For example, for vBucket 1023, the identifier for the uuid statistic + is vb_1023:uuid.

    + +
    + vBucket statistics + + + + Name + Description + + + + + num_items + Number of items in this vBucket. + + + num_tmp_items + Number of temporary items in memory. + + + num_non_resident + Total number of items which are not resident in memory for this vBucket. + + + vb_pending_perc_mem_resident + % memory resident. + + + vb_pending_eject + Number of times item values got ejected. + + + vb_pending_expired + Number of times an item was expired. + + + ht_memory + Memory overhead of the hashtable. + + + ht_item_memory + Total item memory. + + + ht_cache_size + Total size of cache — includes non-resident items. + + + num_ejects + Number of times an item was ejected from memory. + + + ops_create + Number of create operations. + + + ops_update + Number of update operations. + + + ops_delete + Number of delete operations. + + + ops_reject + Number of rejected operations. + + + queue_size + Pending items in disk queue. + + + backfill_queue_size + Items in backfill queue. + + + queue_memory + Memory used for disk queue. + + + queue_age + Sum of disk queue item age in milliseconds. + + + queue_fill + Total enqueued items. + + + queue_drain + Total drained items. + + + pending writes + Total bytes of pending writes. + + + db_data_size + Total size of useful data in the database file on disk, measured in bytes. + + + db_file_size + Total size of the database file on disk (including uncompacted stale data), measured in bytes. + + + high_seqno + The last seqno assigned by this vBucket. + + + purge_seqno + The last seqno purged by the compactor. + + + bloom_filter + Status of the vBucket's bloom filter. + + + bloom_filter_size + Size of the bloom filter bit array. + + + bloom_filter_key_count + Number of keys inserted into the bloom filter. Considers overlapped items as one, so this may not + be accurate at times. + + + uuid + The current vBucket uuid. + + + rollback_item_count + Number of items rolled back. + + + hp_vb_req_size + Number of asynchronous high priority requests. + + + max_cas + Maximum CAS of all items in the vBucket. This is a hybrid logical clock value in nanoseconds. + + + max_cas_str + The vBucket’s current maximum hybrid logical clock (HLC) timestamp. In general, this statistic + shows the value issued to the last mutation, or in certain cases the largest timestamp the vBucket has + received, when the received timestamp is ahead of the local clock. Displayed as a human readable ISO-8601 + timestamp (UTC). + + + total_abs_drift + The accumulated absolute drift for this vBucket’s hybrid logical clock in microseconds. Drift is + always accumulated as an absolute value. + + + total_abs_drift_count + The number of updates applied to total_abs_drift, for the purpose of average or + rate calculations. + + + drift_ahead_threshold_exceeded + How many mutations have been observed with a drift above the + drift_ahead_threshold. + + + drift_ahead_threshold + Threshold at which positive drift will trigger an update to drift_ahead_exceeded, + measured in nanoseconds. + + + drift_behind_threshold_exceeded + How many mutations have been observed with a drift below the + drift_behind_threshold. + + + drift_behind_threshold + The threshold at which positive drift will trigger an update to + drift_behind_exceeded. The value is displayed in nanoseconds as a positive value, but is + converted to a negative value for actual exception checks. + + + logical_clock_ticks + How many times the hybrid logical clock (HLC) has had to increment the logical clock. + + + might_contain_xattrs + True if the vBucket might contain xattrs. True means that xattrs were stored to the vBucket. Note + that the flag does not clear itself if all xattrs were removed. + + + +
    + + + Additional vBucket statistics for ephemeral buckets + + + + Name + Description + + + + + seqlist_count + Number of documents in this vBucket's sequence list. + + + seqlist_deleted_count + Count of deleted documents in this vBucket's sequence list. + + + seqlist_high_seqno + High sequence number in sequence list for this vBucket. + + + seqlist_highest_deduped_seqno + Highest de-duplicated sequence number in sequence list for this vBucket. + + + seqlist_read_range_begin + Starting sequence number for this vBucket's sequence list read range. Marks the lower bound of + possible stale documents in the sequence list. + + + seqlist_read_range_end + Ending sequence number for this vBucket's sequence list read range. Marks the upper bound of + possible stale documents in the sequence list. + + + seqlist_read_range_count + Count of elements for this vBucket's sequence list read range, i.e. end - begin. + + + seqlist_stale_count + Count of stale documents in this vBucket's sequence list. + + + seqlist_stale_value_bytes + Number of bytes of stale values in this vBucket's sequence list. + + + seqlist_stale_metadata_bytes + Number of bytes of stale metadata (key + fixed metadata) in this vBucket's sequence list. + + + +
    +
    + +
    + Options + + vbucket-details options + + + + + + Option + Description + + + + + vbid + vBucket ID. In a standard system this will be between 0 and 1023. If not + provided as part of the command then details for all vBuckets are shown. + + + +
    +

    For common cbstats options, see .

    +
    + +
    + Example + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample vbucket-details 1023 + +

    Response

    + vb_1023: active + vb_1023:backfill_queue_size: 0 + vb_1023:bloom_filter: DOESN'T EXIST + vb_1023:bloom_filter_key_count: 0 + vb_1023:bloom_filter_size: 0 + vb_1023:db_data_size: 4541 + vb_1023:db_file_size: 41051 + vb_1023:drift_ahead_threshold: 5000000000 + vb_1023:drift_ahead_threshold_exceeded: 0 + vb_1023:drift_behind_threshold: 5000000000 + vb_1023:drift_behind_threshold_exceeded: 0 + vb_1023:high_seqno: 10 + vb_1023:hp_vb_req_size: 0 + vb_1023:ht_cache_size: 3910 + vb_1023:ht_item_memory: 3910 + vb_1023:ht_item_memory_uncompressed: 4077 + vb_1023:ht_memory: 2576 + vb_1023:ht_size: 47 + vb_1023:logical_clock_ticks: 0 + vb_1023:max_cas: 1532081263385051136 + vb_1023:max_cas_str: 2018-07-20T10:07:43.385051136 + vb_1023:max_deleted_revid: 0 + vb_1023:might_contain_xattrs: false + vb_1023:num_ejects: 0 + vb_1023:num_items: 10 + vb_1023:num_non_resident: 0 + vb_1023:num_temp_items: 0 + vb_1023:ops_create: 0 + vb_1023:ops_delete: 0 + vb_1023:ops_reject: 0 + vb_1023:ops_update: 0 + vb_1023:pending_writes: 0 + vb_1023:purge_seqno: 0 + vb_1023:queue_age: 0 + vb_1023:queue_drain: 1 + vb_1023:queue_fill: 1 + vb_1023:queue_memory: 0 + vb_1023:queue_size: 0 + vb_1023:rollback_item_count: 0 + vb_1023:total_abs_drift: 0 + vb_1023:total_abs_drift_count: 0 + vb_1023:uuid: 66069026212209 +
    + + + + diff --git a/content/cli/cbstats/cbstats-vbucket-seqno.dita b/content/cli/cbstats/cbstats-vbucket-seqno.dita new file mode 100644 index 0000000000..b040333cb0 --- /dev/null +++ b/content/cli/cbstats/cbstats-vbucket-seqno.dita @@ -0,0 +1,110 @@ + + + + + <cmdname>vbucket-seqno</cmdname> + Provides seqno statistics for vBuckets. + + + +
    + Syntax +

    Request syntax:

    + cbstats host:11210 [common options] vbucket-seqno [vbid] +
    + +
    + Description +

    This command provides details connected to the sequence number (seqno) for the specified vBucket, or for each + vBucket if none is specified.

    + +

    + + + vBucket seqno statistics + + + + Name + Description + + + + + abs_high_seqno + The last seqno assigned by this vBucket. + + + high_seqno + The last seqno assigned by this vBucket, or in case of replica, the last closed checkpoint's end seqno. + + + last_persisted_seqno + The last persisted seqno for the vBucket. + + + purge_seqno + The last seqno purged by the compactor. + + + uuid + The current vBucket uuid. + + + last_persisted_snap_start + The last persisted snapshot start seqno for the vBucket. + + + last_persisted_snap_end + The last persisted snapshot end seqno for the vBucket. + + + +
    +
    + +
    + Options + + vbucket-seqno options + + + + + + Option + Description + + + + + vbid + vBucket ID. In a standard system this will be between 0 and 1023. If not + provided as part of the command then details for all vBuckets are shown. + + + +
    +

    For common cbstats options, see .

    +
    + +
    + Example + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample vbucket-seqno 1023 + +

    Response

    + vb_1023:abs_high_seqno: 10 + vb_1023:high_seqno: 10 + vb_1023:last_persisted_seqno: 10 + vb_1023:last_persisted_snap_end: 10 + vb_1023:last_persisted_snap_start: 10 + vb_1023:purge_seqno: 0 + vb_1023:uuid: 66069026212209 +
    + +     + +     diff --git a/content/cli/cbstats/cbstats-vbucket.dita b/content/cli/cbstats/cbstats-vbucket.dita index 22c152006f..8270ba9957 100644 --- a/content/cli/cbstats/cbstats-vbucket.dita +++ b/content/cli/cbstats/cbstats-vbucket.dita @@ -5,137 +5,28 @@ Provides detailed statistics on a per vbucket granularity.
    Syntax -

    Request syntax for vBucket:

    - cbstats [host]:11210 vbucket -cbstats [host]:11210 vbucket-details [vbid] -cbstats [host]:11210 vbucket-seqno [vbid] -

    These commands are used as follows:

    -
      -
    • vbucket: Lists all available vBuckets and provides information about - their type: active, replica, pending or dead.
      • -
    • vbucket-details: Uses the option vbid to provide - statistics for a specific vBucket.
      • -
    • vbucket-seqno: Uses the option vbid to provide - details connected to the vBucket's sequence number.
      • -
    - - +

    Request syntax:

    + cbstats host:11210 [common options] vbucket
    Description -

    The vbucket command shows whether a vBucket is an active, replica, - pending or dead vBucket.

    - - Notable vBucket-details stats - - - - - - Stat - Description - - - - - db_data_size - Total size of useful data in the database file on disk, measured in - bytes. - - - db_file_size - Total size of the database file on disk (including un-compacted stale data), - measured in bytes. - - - drift_ahead_threshold - Threshold at which positive drift will trigger an update to - drift_ahead_exceeded, measured in nanoseconds. - - - drift_ahead_threshold_exceeded - How many - mutations have been observed with a drift above the - drift_ahead_threshold. - - - drift_behind_threshold - The threshold - at which positive drift will trigger an update to drift_behind_exceeded. The value - is displayed in nanoseconds as a positive value, but is converted to a negative - value for actual exception checks. - - - drift_behind_threshold_exceeded - How many - mutations have been observed with a drift below the - drift_behind_threshold. - - - logical_clock_ticks - How many times - the hybrid logical clock has had to increment the logical clock. - - - max_cas_str - The vBucket’s - current maximum hybrid logical clock timestamp. In general, this statistic shows - the value issued to the last mutation or in certain cases the largest timestamp - the vBucket has received (when the received timestamp is ahead of the local - clock). Displayed as a human readable ISO-8601 timestamp (UTC). - - - - num_items - Total number of items in the vbucket. - - - num_non_resident - Total number of items which are not resident in memory for this - vbucket. - - - total_abs_drift - Accumulated drift observed by the vBucket. Drift is always accumulated as an - absolute value. - - - total_abs_drift_count - How many - updates have been applied to total_abs_drift, for the purpose of average or rate - calculations. - - - -
    +

    This command lists all available vBuckets and provides information about their type: active, replica, pending + or dead.

    - -
    Options - vbucket options - - - - - - Option - Description - - - - - vbid - vBucket ID (in a standard system this will be between 0 and 1023). If not - provided as part of the command then details for all vBuckets are shown. - - - -
    +
    + Options +

    There are no options for this command. For common cbstats options, see .

    -
    ExamplesRequestscbstats 192.0.2.1:11210 vbucketcbstats 192.0.2.1:11210 vbucket-details 115cbstats 192.0.2.1:11210 vbucket-seqno 115

    Responses

    Example - response for - vbucket:

    vb_0: replica +
    + Example + +

    Request

    + cbstats -u Administrator -p password -b beer-sample localhost:11210 vbucket + +

    Response

    + vb_0: replica vb_1: replica vb_10: replica vb_100: active @@ -145,53 +36,9 @@ cbstats [host]:11210 vbucket-seqno [vbid] vb_1003: replica vb_1004: replica vb_1005: replica - ...

    Example - response for vbucket-details - 115:

    vb_115: active - vb_115:bloom_filter: ENABLED - vb_115:bloom_filter_key_count: 0 - vb_115:bloom_filter_size: 95851 - vb_115:db_data_size: 28473 - vb_115:db_file_size: 98395 - vb_115:drift_ahead_threshold: 5000000000 - vb_115:drift_ahead_threshold_exceeded: 0 - vb_115:drift_behind_threshold: 5000000000 - vb_115:drift_behind_threshold_exceeded: 0 - vb_115:high_seqno: 31 - vb_115:ht_cache_size: 44546 - vb_115:ht_item_memory: 44546 - vb_115:ht_memory: 26656 - vb_115:logical_clock_ticks: 9 - vb_115:max_cas: 1484657322073980930 - vb_115:max_cas_str: 2017-01-17T12:48:42.73980930 - vb_115:num_ejects: 4 - vb_115:num_items: 31 - vb_115:num_non_resident: 4 - vb_115:num_temp_items: 0 - vb_115:ops_create: 31 - vb_115:ops_delete: 0 - vb_115:ops_reject: 0 - vb_115:ops_update: 0 - vb_115:pending_writes: 0 - vb_115:purge_seqno: 0 - vb_115:queue_age: 0 - vb_115:queue_drain: 32 - vb_115:queue_fill: 32 - vb_115:queue_memory: 0 - vb_115:queue_size: 0 - vb_115:rollback_item_count: 0 - vb_115:total_abs_drift: 0 - vb_115:total_abs_drift_count: 0 - vb_115:uuid: 265143847629643

    Example - response for vbucket-seqno - 115:

    vb_115:abs_high_seqno: 31 - vb_115:high_seqno: 31 - vb_115:last_persisted_seqno: 31 - vb_115:last_persisted_snap_end: 31 - vb_115:last_persisted_snap_start: 31 - vb_115:purge_seqno: 0 - vb_115:uuid: 265143847629643
    - - + ...     +
    + + diff --git a/content/cli/cbstats/cbstats-vkey.dita b/content/cli/cbstats/cbstats-vkey.dita index 9119e53853..658041869b 100644 --- a/content/cli/cbstats/cbstats-vkey.dita +++ b/content/cli/cbstats/cbstats-vkey.dita @@ -2,16 +2,16 @@ <cmdname>vkey</cmdname> - Provides information on keys. + Provides verification for keys.
    Syntax

    Request syntax:

    - cbstats [hostname]:11210 vkey [keyname] [vbid] + cbstats host:11210 [common options] vkey keyname vbid
    Description

    The following key information is available.

    - + @@ -25,54 +25,77 @@ key_cas - The keys current cas value - KV - - - key_data_age - How long the key has waited for its value to be persisted (0 if - clean) + The key’s current cas value, as a decimal integer. KV key_exptime - Expiration time from the epoch + Expiration time from the epoch, as a decimal integer. KV key_flags - Flags for this key + Flags for this key, as a decimal integer. KV key_is_dirty - If the value is not yet persisted + If the value is not yet persisted. + KV + + + key_is_resident + If the value is resident in memory. KV key_valid - See description below + See description below. * V key_vb_state - The vbucket state of this key + The vBucket state of this key. KV
    -

    key_valid= can have the following responses:

    -
      -
    • this_is_a_bug: Some case we didn’t take care of.
      • -
    • dirty:The value in memory has not been persisted yet.
      • -
    • length_mismatch: The key length in memory doesn’t match the length on the disk.
      • -
    • data_mismatch: The data in memory doesn’t match the data on disk.
      • -
    • flags_mismatch: The flags in memory don’t match the flags on disk.
      • -
    • valid: The key is both on disk and in memory
      • -
    • ram_but_not_disk: The value doesn’t exist yet on disk.
      • -
    • item_deleted: The item has been deleted.
      • -
    +

    * key_valid can have the following responses:

    +
    + +
    this_is_a_bug
    +
    Some case we didn’t take care of.
    +     + +
    dirty
    +
    The value in memory has not been persisted yet.
    +     + +
    length_mismatch
    +
    The key length in memory doesn’t match the length on the disk.
    +     + +
    data_mismatch
    +
    The data in memory doesn’t match the data on disk.
    +     + +
    flags_mismatch
    +
    The flags in memory don’t match the flags on disk.
    +     + +
    valid
    +
    The key is both on disk and in memory.
    +     + +
    ram_but_not_disk
    +
    The value doesn’t exist yet on disk.
    +     + +
    item_deleted
    +
    The item has been deleted.
    +     +
    Options @@ -88,22 +111,59 @@ - KEYNAME + keyname Name of the key. - VBID - vBucket ID. + vbid + vBucket ID. † + +

    † To find the vBucket ID associated with a given key, you can use the cbc-hash + command, which is available as part of the Couchbase C SDK (libcouchbase).

    +
      +
    • For more information on the Couchbase C SDK, see .
      • +
    • For more information on the cbc-hash command, see + Couchbase C Client: + cbc(1) - Couchbase Client Commandline Utility.
      • +
    +     + +

    For common cbstats options, see .

    +
    -
    Example + +
    + Example 1: Find the vBucket ID associated with a given key +

    Request

    - cbstats 10.5.2.54:11210 vkey foo 115 + cbc-hash 21st_amendment_brewery_cafe -U couchbase://localhost/beer-sample -u Administrator -P password + +

    Response

    + 21st_amendment_brewery_cafe: [vBucket=553, Index=1] Server: 10.142.180.102:11210, CouchAPI: http://10.142.180.102:8092/beer-sample +Replica #0: Index=0, Host=10.142.180.101:11210 +
    +
    + Example 2: Get vkey information + +

    Request

    + cbstats localhost:11210 -u Administrator -p password -b beer-sample vkey 21st_amendment_brewery_cafe 553 + +

    Response

    + verification for key 21st_amendment_brewery_cafe + key_cas: 1532081261254082560 + key_exptime: 0 + key_flags: 2 + key_is_dirty: false + key_is_resident: true + key_valid: data_mismatch + key_vb_state: replica
    + diff --git a/content/cli/cbstats/cbstats-workload.dita b/content/cli/cbstats/cbstats-workload.dita index 64a2054e1c..4c870e18fd 100644 --- a/content/cli/cbstats/cbstats-workload.dita +++ b/content/cli/cbstats/cbstats-workload.dita @@ -18,7 +18,7 @@

    The following are the command options:

    workload options - + diff --git a/content/cli/cli.ditamap b/content/cli/cli.ditamap index 11398ccdf5..0a3955c482 100644 --- a/content/cli/cli.ditamap +++ b/content/cli/cli.ditamap @@ -2,7 +2,7 @@ Couchbase Server CLI - + @@ -13,50 +13,57 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -72,6 +79,7 @@ + @@ -83,28 +91,38 @@ + + + + + + + + + + - - + + diff --git a/content/cloud/couchbase-aws-marketplace.dita b/content/cloud/couchbase-aws-marketplace.dita index 65668db1fd..0b3c28d00d 100644 --- a/content/cloud/couchbase-aws-marketplace.dita +++ b/content/cloud/couchbase-aws-marketplace.dita @@ -11,11 +11,11 @@ proceeding. For information on how to do so, see the AWS documentation for Creating an AWS Account. - The command line examples that begin with $ are run in a standard - Linux shell. +
    Deploying Couchbase Enterprise - The CloudFormation templates are provided for testing purposes only and should not be used in production. They are not officially supported by Couchbase. + The CloudFormation templates are provided as a starting point and may + be customized as needed. Note that additional post deployment setup may be required.
    1. Log in to your account on the Amazon Web Services Marketplace, search for @@ -44,8 +44,9 @@ id="image_q4w_yb2_qbb"/>
    2. Review your configuration and then choose Launch CloudFormation to launch your configuration through the AWS CloudFormation console. These templates are provided - for testing purposes only and should not be used in production. They are not officially - supported by Couchbase.
    3. You will be redirected to the AWS CloudFormation Console where you must create a stack. diff --git a/content/cloud/couchbase-azure-best-practices.dita b/content/cloud/couchbase-azure-best-practices.dita index 5e40fc518f..01bea020e3 100644 --- a/content/cloud/couchbase-azure-best-practices.dita +++ b/content/cloud/couchbase-azure-best-practices.dita @@ -13,7 +13,7 @@ href="https://developer.couchbase.com/documentation/server/current/install/pre-install.html" format="html" scope="external">we recommend machines with 4 or more cores for production applications.

      -

      For a majority of applications the ES16 v3 will be a good balance of price and +

      For a majority of applications the DS14 v2 will be a good balance of price and performance.

      We recommend using VM Scale Sets (VMSS) instead of stand alone VMs as it improves reliability and simplifies the addition and removal of nodes.

      diff --git a/content/cloud/couchbase-azure-marketplace.dita b/content/cloud/couchbase-azure-marketplace.dita index a5a6a7ef10..e5ee4f4e23 100644 --- a/content/cloud/couchbase-azure-marketplace.dita +++ b/content/cloud/couchbase-azure-marketplace.dita @@ -9,7 +9,8 @@
    4. Sufficient Core Quota - The Azure Free Trial is a great way to get started. However, it has a core quota of 4 cores per region. Upgrading the trial to a Pay-As-You-Go subscription will increase that quota to 10 cores. Additionally, for a Pay-As-You-Go subscription you can open a support ticket in the Azure Portal to request a quota increase beyond the 10 core default. An Azure Enterprise Agreement (EA) has a 200 core quota by default.
    Deploying Couchbase Enterprise - The templates are provided for testing purposes only and should not be used in production. They are not officially supported by Couchbase. + The templates are provided as a starting point and may be customized as needed. Note that additional post deployment + setup may be required.
    1. Log in to your account on the Microsoft Azure Marketplace.
    2. Search for Couchbase in the Marketplace and select the Couchbase @@ -143,8 +144,8 @@ the sample templates and instructions to deploy the templates that are available here. Note that these templates are provided - for testing purposes only and should not be used in production. They are not - officially supported by Couchbase.

    3. On the final screen, you're presented with the Microsoft Azure and Couchbase End User diff --git a/content/cloud/couchbase-gcp-cloud-launcher.dita b/content/cloud/couchbase-gcp-cloud-launcher.dita index a090d1de93..14278501e0 100644 --- a/content/cloud/couchbase-gcp-cloud-launcher.dita +++ b/content/cloud/couchbase-gcp-cloud-launcher.dita @@ -14,7 +14,8 @@ scope="external">Quotas page in the Cloud Platform Console.
    Deploying Couchbase Enterprise - The templates are provided for testing purposes only and should not be used in production. They are not officially supported by Couchbase. + The templates are provided as a starting point and may be customized as + needed. Note that additional post deployment setup may be required.
    1. Log in to your Google Cloud Platform account and navigate to the Google Cloud Launcher page for Services
      By checking all appropriate check boxes (Data, Query, and Index), you can define what kind of services will be provided on the
      server node that's been added. You can add one, two, or all three services.
      diff --git a/content/clustersetup/adding-nodes.dita b/content/clustersetup/adding-nodes.dita index 5d354adf72..1b6ef94d9a 100644 --- a/content/clustersetup/adding-nodes.dita +++ b/content/clustersetup/adding-nodes.dita @@ -77,10 +77,10 @@ Services Select the appropriate check box to define the kind of services will be provided on the server node that's been added. You can - add one, two, or all four services. The options are: Data Service, GSI Service, - FTS Service, and Query Service. + add one or more services. The options are: Data, Query, Index, Search, Analytics, + and Eventing. @@ -97,90 +97,32 @@ the cluster with the services of your choice.
    Using the CLI -

    Use CLI to add one or more nodes to an existing cluster.

    - - -

    The new nodes must have Couchbase Server installed, and Couchbase Server must be running on each node.

    -

    To add a node: - > couchbase-cli server-add \ - --cluster=localhost:8091 --user cluster-username --password cluster-password \ - --server-add=192.168.0.72:8091 --server-add-username=node-username \ - --server-add-password=node-password \ - --services=data,index,query

    -

    where:

    - -
    - CLI parameters for adding a node - - - - - - Parameter - Description - - - - - --cluster - The IP address of a node in the existing cluster. - - - --user - The username for the existing cluster. - - - --password - The password for the existing cluster. - - - --server-add - The IP address of the node to be added to the cluster. - - - --server-add-username - The username of the node to be added. - - - --server-add-password - The password of the node to be added. - - - -
    -

    If adding was successful, you will see the following response: - SUCCESS: server-add 192.168.0.72:8091

    -

    If you receive a failure message, you will be notified of the type of failure. - You can add multiple nodes in one command by supplying - multiple --server-add command-line options to the command. -

    - Once a node has been successfully added, the Couchbase Server cluster indicates that - a rebalance is required to complete the operation. - -

    Here’s an example of cluster rebalance after adding a new node: Request syntax with - rebalancing:couchbase-cli rebalance - -c [localhost]:8091 - --server-add=[host]:8091 - --server-add-username=[administrator] - --server-add-password=[password]

    -

    You - can cancel the addition of a node to a cluster without having to perform a rebalance - operation. Canceling the operation removes the server from the cluster without having - transferred or exchanged any data, since no rebalance operation took place. You can cancel - the operation using the Couchbase Web Console.

    - - - - - -
    Using the REST APIWith the REST API, you can add nodes to the cluster by - providing the IP address, administrator username, and password as part of the data - payload.

    Use the curl command to add a new node. If you are adding - another service than data, which is the default, specify that service index or n1ql: - > curl -u cluster-username:cluster-password localhost:8091/controller/addNode\ - -d "hostname=192.168.0.68&user=node-username&password=node-password&services=n1ql" -

    +

    + The Couchbase CLI can be used to add one or more nodes to an existing cluster. See the + server-add + command. See also the + rebalance + command, which must be used after the nodes have been added. +

    +
    + +
    + + Using the REST API + + +

    + With the REST API, you can add nodes to the cluster, by + providing the IP address, administrator username, and password as part of the data + payload. See + Adding Nodes to Clusters, + for information. Also, see + Rebalancing Nodes, for + information on how to rebalance the cluster with the REST API, following node-addition. +

    + +
    diff --git a/content/clustersetup/automatic-failover.dita b/content/clustersetup/automatic-failover.dita index 8b079944af..8f969c15ca 100644 --- a/content/clustersetup/automatic-failover.dita +++ b/content/clustersetup/automatic-failover.dita @@ -1,169 +1,242 @@ - Using Automatic Failover The Cluster Manager continuously monitors node availability and can automatically perform a hard failover of a node if it determines that a node is unavailable. + + + Using Automatic Failover + + + A node or group can be failed over automatically, when it either + becomes unavailable or experiences continuous disk-access problems. + + -

    With automatic failover, the Cluster Manager autonomously detects a failed node, verifies - that it failed, and then initiates the hard failover process without user intervention. - Automatic failover does not attempt to fix or even identify the issue that caused the node to - fail. Once an administrator discovers and fixes the issue, he must initiate a rebalance to - return the cluster to a healthy state.

    -

    There are a number of built-in restrictions in Couchbase Server for automatic failover. These - restrictions are in place to strike a balance between a self-healing cluster and the potential - for automatic failover to make a small problem much worse by creating a cascading failure or - data inconsistency. These problems are discussed in Automatic Failover - Considerations. + +

    + + Understanding Automatic Failover + + +

    + Automatic Failover — or auto-failover — can be configured to + fail over a node or group automatically: no + immediate administrator intervention is required. + Specifically, the Cluster Manager autonomously detects and verifies that the node or group is unavailable, + and then initiates the hard failover process. + Auto-failover does not fix or identify problems that may have occurred. Once + appropriate fixes have been applied to the cluster by the administrator, a rebalance is required. + Automatic failover is always hard failover.

    - Automatic failover can be configured at any cluster size or configuration but - will not be triggered until the following conditions are met. -

    Automatic failover is:

    -
      -
    • Disabled by default to prevent Couchbase Server from using it if you didn't enable it - explicitly.
      • -
    • Available only on clusters that contain at least three nodes running the Data service. - This helps prevent a split-brain scenario in the cluster.
      • -
    • Designed to failover a node only if that node is the only one down at a given time. - Combined with the previous restriction, this also prevents a split-brain scenario in the - cluster.
      • -
    • Designed to failover only one node before requiring human intervention. This measure is - introduced to prevent a chain reaction failure of multiple or all nodes in the cluster.
      • -
    • Designed to failover after 120 secondly by default, with a minimum of 5 seconds. This - helps prevent spurious failovers due to network or server slowdowns or interruptions.
      • -
    • Able to failover one node at a time with Rack/Zone Awareness (RZA) enabled. For more - nodes, you will have to fail them manually.
      • -
    • Intended to perform only intra-cluster, so it does not operate with Cross Datacenter - Replication (XDCR).
      • -
    • Available only if the auto-failover counter is at 0.
      • -
    • Not designed to failover the Index Service by default.
      • - - -
    -

    Couchbase Server can be configured to generate an alert (via the Couchbase Web Console or - email) both when a node is automatically failed over or when it has failed but not - automatically failed over.

    + +
    -
    Automatic Failover Timeout -

    By default, there is a 120 second delay before a node will be automatically failed over by - the Cluster Manager. This timeout can be increased to any value up to 3600 seconds, and - decreased to any value above 5 seconds. This 5 second minimum delay is needed by the - software to confirm that the node is indeed down by performing multiple checks of that - node’s status. This helps prevent false alarms, such as failover of a functioning but slow - node or failover due to ephemeral network connection issues. It is strongly recommended - that you do not lower the failover timeout from its default of 120 seconds unless you have - very reliable infrastructure underlying each of your Couchbase Server nodes.

    - - -
    -
    Resetting the Automatic Failover Counter

    After a node has been - automatically failed over, Couchbase Server increments an internal counter that indicates to - the cluster if a node has been failed over. The counter prevents the Cluster Manager from - automatically failing over additional nodes until you identify any issues that caused the - failover and resolve them.

    Reset the automatic failover only:

      -
    • after the node’s issue is resolved, rebalance has occurred, and the cluster was - restored to a fully functioning state.

      or

      • -
    • if the node has the Data - service running and each bucket has additional replicas beyond the existing ones - to handle another node auto-failover.
      • -

    You can reset this counter with the Couchbase Web Console by clicking on the - Reset Quota button as shown below.

    You can also reset the counter using the REST API: - >$ curl -i -u cluster-username:cluster-password http://localhost:8091/settings/autoFailover/resetCount

    Once - the counter has been reset, the Cluster Manager could failover another node of the cluster. So - again, reset the counter only if the criteria for automatic failover are met.
    -
    When Not to Use Automatic Failover -

    Automatic failover is appropriate in most environments. However, it is disabled by default - to ensure that administrators are explicitly aware of the behavior of their cluster.

    -

    There are a few situations where auto-failover would not advisable to use or when the - environment is known to contain instability that could introduce false positives even when - the nodes did not fail:

    +
    + + + Failover Events + + +

    + Auto-failover occurs in response to failover events. Events are of three kinds: +

    +
      -
    • "Flaky" or very slow network between cluster nodes
      • -
    • An issue with the Cluster Manager or operating system
      • +
    • + Node failure. A server-node within the cluster is unavailable (due to a network failure, + out-of-memory problem, or other node-specific issue). + +

      + +

      +
      • + +
    • + Disk read/write failure. Attempts to read from or write to disk on a particular node have resulted in + a significant rate of failure, for longer than a specified time-period. The node is removed by + auto-failover, even though the node continues to be contactable. + +

      + +

      +
      • + +
    • + Group failure. An administrator-defined group of server-nodes within the cluster is unavailable (perhaps + due to a network or power failure that has affected an individual, physical rack of machines, or a specific subnet). +

      + +

      +
      • +
    -

    While very rare, you might also consider turning off auto-failover if the network between - nodes is not the same as the one between the nodes and the client application servers. Such - difference might introduce the possibility of disruption between the nodes of the cluster - without affecting the application’s ability to communicate with the nodes.

    -
    Automatic Failover Considerations -

    Automatically recovering from component failures in any distributed system involves at - least some risks. If you cannot identify the cause of failure, and you do not understand the - load that will be placed on the remaining systems, an automatic failover might actually - cause more problems than it solves. This is why proper sizing and capacity planning of the - cluster must include the possibilities of failures. Some of the situations that might lead - to problems include:

    -
    - -
    Avoiding failover chain-reactions (Thundering Herd)
    -
    Imagine a scenario where a five-node Couchbase cluster is running well and operating - at 80–90% of aggregate capacity with a given load. A single node fails and the software - decides to automatically failover that node. Unless your cluster is properly sized, it - is unlikely that the four remaining nodes will be able to successfully handle the - additional load.

    The result is that the increased load could lead to another node - slowing and appearing to the Cluster Manager to have failed, therefore it too - automatically will be failed over. Such failures can cascade and lead to the eventual - loss of the entire cluster. Clearly, it is better if one fifth of the requests is not - serviced due to a single node failure than to have the entire cluster fail and none of - the requests serviced. To prevent cluster failure, the automatic failover is limited - to a single node in the cluster.

    The partial solution is not to use - auto-failover but to continue cluster operations with the single failed node (alerted - by your monitoring system). You can then add a new server to the cluster to handle the - missing capacity, manually failover the node, and then rebalance the cluster. This way - there is a brief partial outage rather than an entire cluster being disabled. This - solution considered being only a partial because it relies on human intervention or - heavy automation. If the failover is not done promptly, the span in which that portion - of the data is unavailable could be long.

    The best practice is to use automatic - failover, but ensure that the cluster has been sized correctly to handle unexpected - node failures and allow replicas to take over. Continual capacity planning is critical - to a healthy and correct performance of the Couchbase cluster.

    -     -
    -
    - -
    Handling failovers with network partitions (Split Brain Scenario)
    -
    In the event of a network partition or split brain scenario, where the - failure of a network device causes a network to be split, the following restrictions for - the automatic failover are in place at all time:
      -
    • A minimum of three (3) nodes per cluster are required. Such sizing prevents a - 2-node cluster from having both nodes fail in the face of a network partition and - protects the data integrity and consistency.
      • -
    • Occurs only if exactly one (1) node is down. This prevents a network partition - from causing two or more halves of a cluster from failing each other over and - protects the data integrity and consistency.
      • -
    • Occurs only once before requiring administrative action. This prevents cascading - failovers and subsequent performance and stability degradation. In many cases, it is - better to not have access to a small part of the dataset rather than having a - cluster continuously degrade itself to the point of being non-functional.
      • -
    • Implements a 30 second delay when a node fails before it performs an automatic - failover. This prevents transient network issues or slowness from causing a node to - be failed over when it shouldn’t be.
      • -

    If a network partition occurs, automatic failover occurs if and only if - automatic failover is allowed by the specified restrictions. For example, if a single - node is partitioned out of a cluster of five (5), it is automatically failed over. If - more than one (1) node is partitioned off, autofailover does not occur. After that, - administrative action is required for a reset. In the event that another node fails - before the automatic failover is reset, no automatic failover occurs.

    -     -
    - -
    -
    Limitations of Automatic Failover

    In some cases, - automatic failover may take longer than the specified timeout value. This delay can range from - a second up to 75 seconds, depending on which node is being failed over and the nature of the - node failure.

    In particular, significant delays (60 seconds +) are seen when the cluster - manager on the orchestrator node (the node responsible for failing over other nodes) - stops responding to the other nodes. This could be due to networking issues between the - orchestrator and the other nodes, or due to the cluster management process locking up. The - other nodes in the cluster must then all wait for the connections to the orchestrator to - timeout before they can elect a new one. In other situations, such as Data service failure, - the failure can be detected quickly by the other nodes and they do not have to wait to elect - a new orchestrator to perform the failover.

    In general, if any node other than the - orchestrator fails, delays of more than a few seconds in the automatic failover process - should not be experienced.

    +
    + + + Auto-Failover Constraints + + +

    + Couchbase Server applies the following constraints to auto-failover’s handling of events: +

    + + +
      +
    • + Auto-failover occurs only on the occurrence of one event at a time. If multiple events occur + concurrently, or if a second event follows the first prior to auto-failover’s completion, + auto-failover is not triggered. +

      + +

      + +
      • + +
    • + Auto-failover occurs sequentially up to an administrator-specified maximum number of events. After + this maximum number of auto-failovers has been triggered, no further auto-failover occurs, until the + count is manually reset by the administrator. + +

      + +

      + +
      • + +
    • + Auto-failover is never triggered by Couchbase Server when data-loss might result. Therefore, even a single + event may not be responded to; + and an administrator-specified maximum number of events may not be reached. + +

      + +

      + +
      • + +
    • + Auto-failover is never triggered when the resulting number of available nodes within the cluster would not + constitute a majority of the number available before the problem-occurrence. For example, if one of + three nodes or groups were to go offline, auto-failover could be triggered; but if one of two nodes + or groups were to go offline, auto-failover could not be triggered. (This prevents the situation + where functioning nodes or groups, finding they cannot intercommunicate, + fail each other over, and then attempt to continue independently of one another, each + representing itself as the cluster.) + +

      + +

      + +
      • +
    + +

    + Auto-failover can be enabled only when the cluster contains a minimum of three nodes; and should be + configured for groups only when a minimum of three groups have been defined. Additionally, auto-failover + should be configured only when the cluster contains sufficient resources to handle all possible results: + workload-intensity on remaining nodes may increase significantly. Auto-failover is for intra-cluster use + only: it does not work with Cross Datacenter Replication. +

    + +

    + Auto-failover may take significantly longer if the non-available node is that on which the orchestrator + is running, since time-outs must occur before available nodes can elect a new orchestrator-node and thereby + continue. +

    + +

    + See + Email Alerts, for details + on configuring email alerts related to failover. +

    + +

    See , for information on server groups and group failover.

    + +
    + +
    + + + Configuring Auto-Failover + + +

    + Auto-failover is configured by means of parameters that include the following. + +

    + +
      +
    • + Timeout. The number of seconds that must elapse, after a node or group has become unavailable, before auto-failover + is triggered. The default is 120. +

      + +

      +
      • + +
    • + Maximum count. The maximum number of failover events that can occur sequentially and be handled by auto-failover. The + maximum-allowed value is 3, the default is 1. This parameter is available in Enterprise Edition only: in Community + Edition, the maximum number of failover events that can occur sequentially and be handled by auto-failover is always 1. + +

      + +

      +
      • + +
    • + Count. The number of failover events that have occurred. The default value is 0. The value is incremented by + 1 for every automatic-failover event that occurs, up to the defined maximum count: beyond this point, no further + automatic failover can be triggered until the count is reset to 0 through administrator-intervention. + +

      + +

      +
      • + +
    • + Enablement of disk-related automatic failover; with corresponding time-period. Whether automatic + failover is enabled to handle continuous + read-write failures. If it is enabled, + a number of seconds can also be specified: this is the length of a constantly recurring time-period against which + failure-continuity on a particular node is evaluated. If at least 60% of + the most recently elapsed instance of the time-period has consisted of continuous failure, failover is automatically triggered. + The default value for the enablement of disk-related automatic failover is false. The default value for the + corresponding time-period is 120. + This parameter is available in Enterprise Edition only. +

      + +

      +
      • + +
    • + Group failover enablement. Whether or not groups should be failed over. A group failover is considered to + be a single + event, even if many nodes are included in the group. The default value is false. This parameter is available in + Enterprise Edition only. +

      + +

      +
      • + +
    + + +

    + For more detailed information, + see the documentation provided for specifying + Node Availability with + Couchbase Web Console UI, + for + Managing Auto-Failover + with the REST API, and + setting-autofailover + with the CLI. +

    +
    diff --git a/content/clustersetup/bucket-flush.dita b/content/clustersetup/bucket-flush.dita index d933f9ec26..1222e49e04 100644 --- a/content/clustersetup/bucket-flush.dita +++ b/content/clustersetup/bucket-flush.dita @@ -1,6 +1,6 @@ - + Flush a Bucket @@ -19,7 +19,7 @@

    - To edit an existing bucket-configuration, access the Couchbase Web Console, and left-click on the Buckets + To edit an existing bucket-configuration, access Couchbase Web Console, and left-click on the Buckets tab, in the vertical navigation-bar at the left-hand side.

    @@ -33,7 +33,7 @@

    - +

    @@ -44,12 +44,12 @@

    This displays the Edit Bucket Settings dialog, which permits changes to be made to a subset of existing settings. All the settings contained here are described in detail for - the Add Data Bucket dialog, in the section - Create a Bucket + the + Add Data Bucket dialog, in the section + Create a New Bucket

    - Left-click on the Show advanced bucket settings tab. This causes the Add + Left-click on the Show advanced bucket settings tab. This causes the + Add Data Bucket dialog to expand vertically; and thereby display additional information. Navigate to the bottom of the expanded dialog, and locate the Flush panel. This provides a checkbox, the checking of which enables flushing for the current bucket: @@ -61,7 +61,7 @@

    Note that flushing can also be enabled during bucket-creation. See - Create a Bucket + Initialize the Cluster for details.

    @@ -106,8 +106,8 @@

    See - Authorization, - for information on establishing users and roles. + Creating + and Managing Users with the UI, for information on establishing users and roles.

    @@ -120,7 +120,7 @@

    You can also enable flushing by means of the CLI command - bucket-flush, + bucket-flush, and the REST API method rest-bucket-flush.

    diff --git a/content/clustersetup/bucket-setup.dita b/content/clustersetup/bucket-setup.dita index a5ac3df944..faf3ebf345 100644 --- a/content/clustersetup/bucket-setup.dita +++ b/content/clustersetup/bucket-setup.dita @@ -31,7 +31,7 @@ of all three types are supported simultaneously by Couchbase Server. When you create a bucket, the type you choose depends on the data it is to contain, the source of that data, and the context in which the data will be accessed by users and applications. For a complete architectural description of bucket-types, see - Buckets. + Buckets.

    diff --git a/content/clustersetup/change-settings-bucket.dita b/content/clustersetup/change-settings-bucket.dita index 2f25df4817..2a1cd443b6 100644 --- a/content/clustersetup/change-settings-bucket.dita +++ b/content/clustersetup/change-settings-bucket.dita @@ -34,7 +34,7 @@

    - +

    @@ -104,6 +104,28 @@ +

  • + Bucket Max Time-to-Live: The maximum time a document + can exist, following its creation within this bucket, before being deleted. Can be changed for + a Couchbase or Ephemeral bucket only. A modified setting applies only to documents that will + be created or modified subsequently. + +

    + +

    +
    • + +
  • + Compression Mode: Whether and how compression is applied to data within the bucket. For + information on available modes, and the effect of changing the mode of an existing + bucket, see + Compression. + +

    + +

    +
    • +
  • Ejection Method: The ejection policy used by a bucket. Can be changed for a Couchbase bucket only. Note that changing the ejection-policy forces @@ -183,7 +205,7 @@

    You can change bucket-settings using the CLI command - bucket-edit; or the + bucket-edit; or the Bucket REST API.

    • @@ -192,6 +214,6 @@ - + diff --git a/content/clustersetup/create-bucket.dita b/content/clustersetup/create-bucket.dita index 9e408bed2f..e11aea3f71 100644 --- a/content/clustersetup/create-bucket.dita +++ b/content/clustersetup/create-bucket.dita @@ -94,7 +94,7 @@ Three Bucket Type options are displayed: which are Couchbase, Memcached, and Ephemeral. For information on the differences between these bucket-types, see - Buckets. + Buckets.

    @@ -144,6 +144,34 @@ +

  • +

    + Bucket Max Time-To-Live: If the Enable + checkbox is checked, the integer specified in the seconds + field determines + the maximum time a document + can exist, following its creation within this bucket, before being deleted. The maximum + time that can be specified is 2147483648 (68.096 years). The setting is + applied to all documents created after the setting is itself established. +

    + +

    + For detailed information, see + Expiration. +

    + +
    • + +
  • +

    + Compression Mode: Controls whether and how compression is + applied to data within the bucket. + For detailed information, see + Compression. + +

    +
    • +

  • @@ -161,7 +189,7 @@

    Note that you can also set the conflict resolution method using the CLI command, or the + href="../cli/cbcli/couchbase-cli-bucket-create.html" format="html"/> command, or the REST API.

    • @@ -174,7 +202,7 @@ If Value-only is selected, only data is ejected. Generally, Value-only ejection favors performance at the expense of memory; and Full ejection vice versa. See - , + , for more information. Note that ejection in the context of a Couchbase bucket means removal from memory, with continued persistence on disk. @@ -196,7 +224,7 @@ Specifying High may result in faster processing for the current bucket's tasks. However, the specification only makes a difference when there is more than one bucket defined for the cluster, and when those buckets are assigned different priority-values. See - , for further information. + Data Service, for further information.

    @@ -265,7 +293,9 @@

    - The settings Conflict Resolution, and + The settings Conflict Resolution, + Bucket Max Time-to-Live, + Compression Mode, and Flush are identical in functionality for both Ephemeral and Couchbase buckets.

    @@ -395,7 +425,7 @@

    You can also create a bucket using either the Couchbase Server CLI command - bucket-create, or the REST + bucket-create, or the REST API command rest-bucket-create.

    diff --git a/content/clustersetup/delete-bucket.dita b/content/clustersetup/delete-bucket.dita index dddc04e2d6..deddef47ae 100644 --- a/content/clustersetup/delete-bucket.dita +++ b/content/clustersetup/delete-bucket.dita @@ -40,7 +40,7 @@

    - +

    @@ -76,7 +76,7 @@

    You can also delete a bucket by using the CLI command - bucket-delete, + bucket-delete, or the REST API method rest-bucket-delete.

    diff --git a/content/clustersetup/failover.dita b/content/clustersetup/failover.dita index af07043f17..f269d50940 100644 --- a/content/clustersetup/failover.dita +++ b/content/clustersetup/failover.dita @@ -4,13 +4,15 @@ Failing over a NodeFailover is the process in which a node of a Couchbase cluster is removed quickly as opposed to a regular removal and rebalancing. -

    There are three types of failovers: graceful, hard, and +

    There are three types of failover: graceful, hard, and automatic.

    - To fully understand how failover works, see to learn how - Couchbase partitions data and how database services are distributed across a Couchbase - cluster. + To fully understand how failover works, see + and + + to learn how + data and services are distributed across a Couchbase + Cluster.
    Graceful failover
    @@ -38,12 +40,10 @@ when a node is unavailable and then initiate a hard failover.
    -

    Keep in mind that each of the four Couchbase services; Index, Query, Data and Full Text - Search (FTS) react slightly differently due to their roles. For example, if you have the - services separated on different nodes using Multi-Dimensional Scaling (MDS), you can failover and recover nodes separately. If - you are mixing the services on the same nodes, it is slightly more complex, but not much. - Failover in regards to all four services will be discussed in this section.

    +

    Keep in mind that each of the Couchbase Services reacts differently to failover. For example, + if you have the services separated on different nodes, you can failover and recover nodes + separately. If you are mixing the services on the same nodes, it is slightly more complex. + Failover regarding all services will be discussed in this section.

    Choosing a Failover Solution when Nodes are Unhealthy

    As a node failover has the potential to reduce the performance of your cluster, diff --git a/content/clustersetup/file-locations.dita b/content/clustersetup/file-locations.dita index 07d2d395be..83478a1ca4 100644 --- a/content/clustersetup/file-locations.dita +++ b/content/clustersetup/file-locations.dita @@ -71,7 +71,7 @@ to the node's addition to the cluster. For this purpose, you can use the Configure Server screen of the Couchbase Web Console, the REST API, - or the CLI. + or the CLI.

    diff --git a/content/clustersetup/flush-bucket.dita b/content/clustersetup/flush-bucket.dita index 008b7cd50f..9f68a06629 100644 --- a/content/clustersetup/flush-bucket.dita +++ b/content/clustersetup/flush-bucket.dita @@ -1,6 +1,6 @@ - + Flush a Bucket @@ -19,7 +19,7 @@

    - To edit an existing bucket-configuration, access the Couchbase Web Console, and left-click on the Buckets + To edit an existing bucket-configuration, access Couchbase Web Console, and left-click on the Buckets tab, in the vertical navigation-bar at the left-hand side.

    @@ -33,7 +33,7 @@

    - +

    @@ -120,7 +120,7 @@

    You can also enable flushing by means of the CLI command - bucket-flush, + bucket-flush, and the REST API method rest-bucket-flush.

    diff --git a/content/clustersetup/hard-failover.dita b/content/clustersetup/hard-failover.dita index 622f606770..839cf2b0af 100644 --- a/content/clustersetup/hard-failover.dita +++ b/content/clustersetup/hard-failover.dita @@ -11,8 +11,8 @@ initiated, it tells the Cluster Manager to remove the node from active use, regardless of what is occurring, and then to start the procedures necessary for the other nodes in the cluster to take over for that ejected node. The Cluster Manager performs slightly different actions for - each service (Data, Index, Query, Full Text Search) running on the node being failed over. We - will discuss these service-specific actions later in this section.

    + each service running on the node being failed over. We + will discuss some of these service-specific actions later in this section.

    You can perform hard failover on any node in the cluster.

    Do not use hard failover for regular planned maintenance. For that purpose see the explanations provided in or @@ -99,31 +99,30 @@ queries.

    -
    Hard Failover Effect on Nodes Running the Full Text Search (FTS) Service +
    Hard Failover Effect on Nodes Running the Search Service

    Full text indexes are partitioned and automatically distributed across nodes if multiple - nodes are running the FTS service. When a node running the FTS service is failed over, it - stops taking traffic. If there are no other nodes running FTS, all full text index building - stops and full text queries fail. If there is at least one other node running FTS, other + nodes are running the Search service. When a node running the Search service is failed over, it + stops taking traffic. If there are no other nodes running Search, all full text index building + stops and full text queries fail. If there is at least one other node running Search, other nodes continue responding to queries and return partial results, which your application may choose to accept or not, depending on your requirements.

    When the administrator rebalances the cluster, it performs multiple operations depending on - the level of redundancy you have designed for the FTS service by configuring the replicas + the level of redundancy you have designed for the Search service by configuring the replicas for the full text index:

      -
    • If configured, the FTS service promotes the replicas to active on the remaining nodes of +
    • If configured, the Search service promotes the replicas to active on the remaining nodes of the cluster.
      • -
    • If not configured, the FTS service rebuilds the +
    • If not configured, the Search service rebuilds the indexes on the remaining nodes of the cluster using stored index definitions.
    -

    The FTS service does not perform Delta Node Recovery.

    +

    The Search service does not perform Delta Node Recovery.

    Returning the Cluster to a Stable State

    If or when the failed node is repaired and ready, it can be added back to the cluster via - Delta Node Recovery, Full Recovery, or an entirely new node + Delta Node Recovery, Full Recovery, or an entirely new node could be added.

    • If Delta Node Recovery is an option, the Cluster Manager recognizes this node as a @@ -192,15 +191,15 @@ configured in the bucket/cluster. For example, if you require the ability to failover two nodes, you must configure two replicas for each bucket. Failure to do so will result in a loss of data. Simply put, do not failover more nodes than there are - replicas configured for all buckets.

      The exception to the above rule is when the - Rack/Zone Awareness (RZA) feature is configured. RZA allows designating which nodes - are in a server rack in a data center, different VM hosts or availability zones in a - cloud hosting provider. It ensures that the replica vBuckets for the nodes residing in - Rack A are never in Rack A. When using RZA, it is safe to failover an entire rack’s - worth of Couchbase nodes without data loss or interrupting your application: because - the other racks contain nodes with the replicas. For more information see Rack Zone Awareness. -

    + replicas configured for all buckets.

    The exception to the above rule is when Server + Group Awareness is enabled. Server Group Awareness allows you to specify which + server nodes are in a server rack in a data center, on different VM hosts, or different + availability zones in the cloud. It ensures that the replica vBuckets for the nodes + residing in Rack A are never stored in Rack A. When using Server Group Awareness, it is + safe to failover an entire rack’s worth of Couchbase nodes without data loss or + interrupting your application, because the other racks contain the nodes with the + replicas.
    diff --git a/content/clustersetup/logging.dita b/content/clustersetup/logging.dita new file mode 100644 index 0000000000..c2edc8e9b5 --- /dev/null +++ b/content/clustersetup/logging.dita @@ -0,0 +1,334 @@ + + + + + Logging + + + The Logging facility allows a record to be maintained of important events + that occur on Couchbase Server. + + + + +
    + + + Logging Overview + + +

    + The Couchbase Logging facility records important events, and saves the details to log files, on disk. Additionally, + subsets of information are provided on the Logs screen, in Couchbase Web Console. This may + appear as follows: +

    + +

    + +

    + +

    + By default, on Linux systems, log files are saved to /opt/couchbase/var/lib/couchbase/logs; on MacOS, + to /Users/username/Library/Application Support/Couchbase/var/lib/couchbase/logs; + and on Windows, to C:\Program Files\Couchbase\Server\var\lib\couchbase\logs. +

    + +
    + +
    + + + Collecting Information + + +

    + On each node within a Couchbase Server-cluster, logging is performed continuously. A subset of + the results can be reviewed + in the Couchbase Web Console Logs screen; while all details are saved to + the logs directory, + as described above. + (Note that the logs directory may include + audit.log. This is a special log file, used to manage cluster-security, and + is handled separately from the other log files. The information provided throughout the + remainder of this page — on collecting, uploading, + and redacting — does not apply to audit.log. For information on audit.log, + see + Auditing.) +

    + +

    + Additionally, explicit logging can be performed by the user. This allows + comprehensive and fully updated + information to be generated as required. The output includes everything currently + on disk, together with additional data that is gathered in real time. Explicit logging can either be performed + for all nodes in the cluster, or for one or more individual nodes. The results are saved as zip + files: each zip file contains the log-data generated for an individual node. +

    + +

    + Explicit logging can be performed by means + of the Couchbase CLI utility cbcollect_info. + The documentation for this utility, provided + here, includes + a complete list of the log files that can be created, and a description of the contents of each. +

    + +

    + Explicit logging can also be performed by means of Couchbase Web Console: on the Logs page, + left-click on the + Collect Information tab, located near the top. +

    + +

    + +

    + +

    + This brings up the Collect Information screen: +

    + +

    + +

    + +

    + This allows logs and diagnostic information to be collected either from all or from selected nodes + within the cluster. It also allows a Log Redaction Level to be specified (this is + described in + Applying Redaction, + below). The Specify custom temp directory checkbox can be checked to specify the absolute pathname + of a directory into which data is temporarily saved, during the collection process. + The Specify custom destination directory can be checked to specify the absolute pathname + of a directory + into which the competed zip files are saved. + +

    + +

    + The Upload to Couchbase checkbox is described in + Uploading Log Files, + below. +

    + +

    + To start the collection-process, left-click on the Start Collection + button. A notification is displayed, indicating that the collection-process is running. When + the process has completed, the following information is displayed: +

    + +

    + +

    + +

    + As this indicates, a set of log files has been created for + each node in the cluster. Each file is + saved as a zip file in the stated temporary location. +

    + +
    + +
    + + + Uploading Log Files + + +

    + Log files can be uploaded to Couchbase, for inspection by Couchbase Support. +

    + +

    + For information on performing upload at the command-prompt, see + cbcollect_info. To upload + by means of Couchbase Web Console, before starting the collection-process, check the + Upload to Couchbase checkbox: +

    + +

    + +

    + +

    + The display changes to the following: +

    + +

    + +

    + +

    + The dialog now features an Upload to Host field, which contains the server-location to which the customer-data + is uploaded. Fields are also provided for Customer Name (required) and Ticket Number (optional). + The Upload Proxy field optionally takes the hostname of a remote system, which contains + the directory specified by the pathname. +

    + +

    + Left-click on the Start Collection button. When collection is complete, a notification + provides the URL of the uploaded zip file. +

    + +
    + +
    + + + Understanding Redaction + + +

    + Optionally, log files can be redacted. This means that user-data, + considered to be private, is removed. Such data includes: +

    + +
      +
    • + Key/value pairs in JSON documents +
      • + +
    • + Usernames +
      • + +
    • + Query-fields that reference key/value pairs and/or usernames +
      • + +
    • + Names and email addresses retrieved during product registration +
      • + +
    • + Extended attributes +
      • + +
    + +

    + This redaction of user-data is referred to as partial redaction. (Full redaction, which + will be available in a forthcoming + version of Couchbase Server, additionally redacts meta-data.) +

    + +

    + In each modified log file, + encrypted text (achieved with SHA1) + is substituted for redacted text. For example, the following log-file fragment displays + private data — a Couchbase username: +

    + + 0ms [I0] {2506} [INFO] (instance - L:421) Effective connection string: +couchbase://127.0.0.1?username=Administrator&console_log_level=5&;. +Bucket=default + +

    + The redacted version of the log file might appear as follows: +

    + + 0ms [I0] {2506} [INFO] (instance - L:421) Effective connection string: +<UD>e07a9ca6d84189c1d91dfefacb832a6491431e95</UD>. +Bucket=<UD>e16d86f91f9fd0b110be28ad00e348664b435e9e</UD> + +

    + Note that redaction may eliminate some parameters containing non-private data, as well as all + parameters containing private. +

    + +

    + Note also that redaction of log files may have one or both of the following consequences: +

    + +
      +
    • Logged issues will be found harder to diagnose, by both the user and Couchbase Support. + +

      + +

      +
      • + +
    • + Log-collection is significantly more time-consumptive, since redaction is performed at collection-time. + +

      + +

      +
      • +
    + +
    + +
    + + + Applying Redaction + + +

    + Redaction of log files saved on the cluster can be applied as required, when performing explicit logging, by + means of either cbcollect_info or the Logs facility of Couchbase Web Console. + +

    + +

    + For information on performing explicit logging with redaction at the + command-prompt, see + cbcollect_info. +

    + +

    + To perform explicit logging with redaction + by means of Couchbase Web Console, before starting the collection-process, + access the Log Redaction Level panel, on the Collect Information screen. + This features two radio-buttons, labelled None and Partial Redaction. Make sure the + Partial Redaction radio-button is selected. Guidance on redaction is displayed below it: +

    + +

    + +

    + +

    + Left-click on the Start Collection button. A notification explains that the + collection-process is now running. When + the process has completed, a further notification appears, specifying the location (local or remote) of each + created zip file. Note that, when redaction has been specified, two zip files are provided for each node: one + file containing redacted data, the other unredacted data. +

    + +
    + +
    + + + Redacting Log Files Outside the Cluster + + +

    + Certain Couchbase technologies — such as cbbackupmgr, the SDK, connectors, and + Mobile — create log files saved outside the Couchbase Cluster. These can be redacted by + means of the command-line tool cblogredaction. Multiple log files can be + specified simultaneously. Each file must be specified as plain text. Optionally, the salt to be used + in encryption can be automatically generated. +

    + +

    + For example: +

    + + $ cblogredaction /Users/username/testlog.log -g -o /Users/username -vv +2018/07/17T11:27:06 WARNING: Automatically generating salt. This will make it difficult to cross reference logs +2018/07/17T11:27:07 DEBUG: /Users/username/testlog.log - Starting redaction file size is 19034284 bytes +2018/07/17T11:27:07 DEBUG: /Users/usernae/testlog.log - Log redacted using salt: <ud>COeAtexHB69hGEf3</ud> +2018/07/17T11:27:07 INFO: /Users/username/testlog.log - Finished redacting, 50373 lines processed, 740 tags redacted, 0 lines with unmatched tags + +

    + For more information, see the corresponding man page, or run the command with the --h (help) + option. +

    + +
    + + + +     diff --git a/content/clustersetup/manage-cluster-intro.dita b/content/clustersetup/manage-cluster-intro.dita index 23e276e95a..dde7eb12cd 100644 --- a/content/clustersetup/manage-cluster-intro.dita +++ b/content/clustersetup/manage-cluster-intro.dita @@ -113,6 +113,6 @@
    - + diff --git a/content/clustersetup/manage-groups.dita b/content/clustersetup/manage-groups.dita index e6eb91df5e..ae8be9d5da 100644 --- a/content/clustersetup/manage-groups.dita +++ b/content/clustersetup/manage-groups.dita @@ -1,68 +1,342 @@ - Manage Server GroupsYou can create logical groupings of servers on a cluster where each server group - physically belongs to a rack or availability zone. + + Managing Server Groups + + + Nodes can be assigned to groups, in order to protect a cluster from + large-scale infrastructure failure. + + -

    This section describes how to manage server groups through the Couchbase Web Console. Server - and server groups can also be managed through the Couchbase command-line interface (CLI) and - REST API.

    -

    - By default, when a Couchbase cluster is initialized, Group 1 is created. -

    + Server groups and the Server Group Awareness feature are only available + in Couchbase Server Enterprise Edition. + +
    + + Creating and Maintaining Groups + + +

    + Groups can be administrator-defined to contain a number of the nodes within a + Couchbase Cluster, and thereby protect the cluster against large-scale infrastructure + failure. An explanation of Server Group Awareness, and the effect it has on the + distribution of vBucket replicas throughout a cluster, is provided in .

    + When you initialize a new Couchbase Server cluster, the first node is automatically + placed in a server group named Group 1. Once you create additional server groups, the + Assign Group field becomes available when adding new server nodes + to the cluster. + +

    + To manage groups, left-click on the Servers tab, in the left-hand + navigation bar. This brings up the Servers screen. Now left-click + on the Groups tab, at the upper right: +

    + +

    + +

    + +

    + This brings up the Server Groups screen, which might initially appear as + follows: +

    + +

    + +

    + +

    + This display indicates that the cluster currently contains three servers. Note that the list + of servers is headed by the name Group 1: this is because by default, Couchbase + Server puts each new server into a group with this name. To change the name, left-click + on the edit name tab, adjacent to the name: +

    + +

    + +

    + +

    + This brings up the Edit Group Name dialog: +

    + +

    + +

    + +

    + If you wish to change the group name, edit it + within the interactive text-field, and then left-click on the Rename Group button. +

    + +
    -
    Adding a Server Group -

    To add a new group using the Web Console:

    + +
    + + Add a Group + + +

    + To add a new group, proceed as follows: +

    +
      -
    1. From the Couchbase Web Console > Servers > click - Groups located on the top right side.
      2. -
    2. Click Add Group to add a new group.
      3. -
    3. Enter the group name in the Group Name field and click - Create.
      4. + +
    4. + Left-click on the Add Group tab, which is located at the upper right of + the Server Groups screen: + +

      + +

      + +

      + This brings up the Add Group dialog: +

      + +

      + +

      + +

      + +

      + +
      5. + +
    5. + Access the interactive text-field of the Add Group dialog, and enter + an appropriate name for the new group. Then, left-click on the Add Group + button, to the lower right of the dialog. The new group is added, the dialog + disappears, and the + Server Groups screen is now displayed as follows: + +

      + +

      + +

      +

      +
    -

    Once the information is entered successfully, a new group is added in the Server +

    + Once the information is entered successfully, a new group is added in the Server Groups page.

    -
    - Editing a Server Group -

    To edit a server group name using the Web Console:

    -
      -
    1. From the Couchbase Web Console > Servers > Groups page, select - the group that you want to edit.
      2. -
    2. Click edit name to change the group name.
      3. -
    3. In the Edit Group window, click Rename. -

      The selected group name is updated.

      4. -
    -
    - Deleting a Server Group -

    Server groups are deleted by removing all nodes from the server group and then deleting the - server group.

    -

    To delete a server group using the Web Console:

    -
      -
    1. From the Couchbase Web Console > Servers > Groups page, select - the group that you want to delete.
      2. -
    2. Click delete group to delete the group.
      3. -
    3. In the Confirm Delete Group window, click Delete - Group.

      The selected group is deleted.

      4. -
    -
    - Moving Servers Between Server Groups -

    Servers can be moved between server groups from the Server Groups - page.

    -

    To move a server from one group to another using the Web Console:

    +
    + +
    + + + Move a Server Between Groups + + +

    + Proceed as follows: +

    +
      -
    1. From the Couchbase Web Console > Servers > Groups page, select - the server that you want to move to another group.
      2. -
    2. Select move to pull-down menu to select the new group name. Once - you select the new group, the pending move to <Group Name> message - appears.
      3. -
    3. Click Apply Changes to confirm the server move.
      4. -
    4. Make sure to Rebalance from the Servers - page. For more information on how to rebalance, see .
      5. +
    5. + On the Server Groups screen, access the move to tab at the + right-hand side of the row for the server you wish to move. A pop-up menu + appears, listing the available groups to which the server can be moved. + +

      + +

      + +

      + In this case, there is only one available group to which the server can be moved, + which is Group 2. Select this group. A pending move notification now + appears. +

      + +

      + +

      + +

      + +

      +
      6. + +
    6. + Toward the upper right of the Server Groups screen a Reset tab + and an Apply Changes button have become available. If you left-click on + the Reset tab, the pending move is cancelled, and the selected server + remains in its current group. Left-click on the Apply Changes button to + complete the move. + +

      + +

      + +
      7. + +
    7. + Return to the Servers screen. Note that a rebalance is now recommended, due to + the change you have made: + +

      + +

      + +

      + Left-click on the Rebalance button, to start the rebalance. +

      + +
      8. +
    + +
    + +
    + + + Delete a Group + + +

    + To delete a group, first remove all nodes from the group — either by moving + them to other groups, or by removing them entirely from the cluster. Then, delete + the group. To delete a group by removing servers, proceed as follows. +

    + +
      +
    1. + Access the Servers screen, and left-click on the row for each server you need + to remove, in order to delete their group. This expands the row-display. The + Remove button appears at the lower right: + +

      + +

      +

      + +

      +
      2. + +
    2. + Left-click on the Remove button, to remove the server. A confirmation notification appears: + +

      + +

      + +

      + Left-click on the Remove Server button, to confirm. +

      + +

      + A REMOVAL pending rebalance notification now appears on the row. A Cancel Remove button is + also provided, to allow cancellation: +

      + +

      + +

      + +

      + +

      + +
      3. + +
    3. + Left-click on the Rebalance button to perform a rebalance, and thereby complete the server-removal + process: + +

      + +

      + +

      + At the conclusion of the rebalance, the server has been removed. +

      + +

      + +

      + +

      + Perform server-removal in this way for every server in the group to be deleted, until the group contains + no servers. +

      + +
      4. + +
    4. + Access the Server Groups screen, by left-clicking the Groups button, located at the + upper right of the Servers screen. The group that now contains no servers is represented + by a row that displays a notification: + +

      + +

      + +

      + +

      +
      5. + +
    5. + Left-click on the delete group tab: + +

      + +

      + +

      + A confirmation dialog now appears: +

      + +

      + +

      + +

      + Left-click on the Delete Group button. The group is deleted, and now longer appears as a row on + the Server Groups screen. +

      + +

      + +

      +
      6. +
    + +
    + +
    + + + Assign a Group when Adding a Server + + +

    + When a server is added to a cluster, the server can be assigned to an + existing group. The Add Server Dialog appears as follows: +

    + +

    + +

    + +

    + Select one of the existing groups from the controls that are located at the right of the Assign Group field. +

    + +

    + +

    +
    diff --git a/content/clustersetup/picts/accessBucketTab.png b/content/clustersetup/picts/accessBucketTab.png index 59680df3ed..69f40a1592 100644 Binary files a/content/clustersetup/picts/accessBucketTab.png and b/content/clustersetup/picts/accessBucketTab.png differ diff --git a/content/clustersetup/picts/addDataBucketDialogExpanded.png b/content/clustersetup/picts/addDataBucketDialogExpanded.png index ce85bbfa87..d266e37fe5 100644 Binary files a/content/clustersetup/picts/addDataBucketDialogExpanded.png and b/content/clustersetup/picts/addDataBucketDialogExpanded.png differ diff --git a/content/clustersetup/picts/addDataBucketDialogExpandedForEphemeral.png b/content/clustersetup/picts/addDataBucketDialogExpandedForEphemeral.png index 7137d7cc30..88bf419a89 100644 Binary files a/content/clustersetup/picts/addDataBucketDialogExpandedForEphemeral.png and b/content/clustersetup/picts/addDataBucketDialogExpandedForEphemeral.png differ diff --git a/content/clustersetup/picts/addDataBucketDialogExpandedForMemcached.png b/content/clustersetup/picts/addDataBucketDialogExpandedForMemcached.png index 7da620f947..dc13338e02 100644 Binary files a/content/clustersetup/picts/addDataBucketDialogExpandedForMemcached.png and b/content/clustersetup/picts/addDataBucketDialogExpandedForMemcached.png differ diff --git a/content/clustersetup/picts/addDataBucketDialogInitial.png b/content/clustersetup/picts/addDataBucketDialogInitial.png index 6a2310f540..f434d3acc5 100644 Binary files a/content/clustersetup/picts/addDataBucketDialogInitial.png and b/content/clustersetup/picts/addDataBucketDialogInitial.png differ diff --git a/content/clustersetup/picts/bucketsViewInitial.png b/content/clustersetup/picts/bucketsViewInitial.png index 68cc1cda11..e3ebc390bb 100644 Binary files a/content/clustersetup/picts/bucketsViewInitial.png and b/content/clustersetup/picts/bucketsViewInitial.png differ diff --git a/content/clustersetup/picts/bucketsViewInitialEdit.png b/content/clustersetup/picts/bucketsViewInitialEdit.png new file mode 100644 index 0000000000..e16cc98eba Binary files /dev/null and b/content/clustersetup/picts/bucketsViewInitialEdit.png differ diff --git a/content/clustersetup/picts/bucketsViewWithExpandedBucketRow.png b/content/clustersetup/picts/bucketsViewWithExpandedBucketRow.png index 72d23edb53..5276049396 100644 Binary files a/content/clustersetup/picts/bucketsViewWithExpandedBucketRow.png and b/content/clustersetup/picts/bucketsViewWithExpandedBucketRow.png differ diff --git a/content/clustersetup/picts/collectInfo.png b/content/clustersetup/picts/collectInfo.png new file mode 100644 index 0000000000..1ac9ac6410 Binary files /dev/null and b/content/clustersetup/picts/collectInfo.png differ diff --git a/content/clustersetup/picts/collectInformationComplete.png b/content/clustersetup/picts/collectInformationComplete.png new file mode 100644 index 0000000000..f0ac6b2765 Binary files /dev/null and b/content/clustersetup/picts/collectInformationComplete.png differ diff --git a/content/clustersetup/picts/collectInformationInProgress.png b/content/clustersetup/picts/collectInformationInProgress.png new file mode 100644 index 0000000000..a6e8df7aac Binary files /dev/null and b/content/clustersetup/picts/collectInformationInProgress.png differ diff --git a/content/clustersetup/picts/collectInformationScreen.png b/content/clustersetup/picts/collectInformationScreen.png new file mode 100644 index 0000000000..aaecd7d176 Binary files /dev/null and b/content/clustersetup/picts/collectInformationScreen.png differ diff --git a/content/clustersetup/picts/failoverMultipleNodesClickButton.png b/content/clustersetup/picts/failoverMultipleNodesClickButton.png new file mode 100644 index 0000000000..9f6c6a953a Binary files /dev/null and b/content/clustersetup/picts/failoverMultipleNodesClickButton.png differ diff --git a/content/clustersetup/picts/failoverMultipleNodesClusterInitialScreen.png b/content/clustersetup/picts/failoverMultipleNodesClusterInitialScreen.png new file mode 100644 index 0000000000..f1c9e208d8 Binary files /dev/null and b/content/clustersetup/picts/failoverMultipleNodesClusterInitialScreen.png differ diff --git a/content/clustersetup/picts/failoverMultipleNodesDialog.png b/content/clustersetup/picts/failoverMultipleNodesDialog.png new file mode 100644 index 0000000000..3be62e1a1c Binary files /dev/null and b/content/clustersetup/picts/failoverMultipleNodesDialog.png differ diff --git a/content/clustersetup/picts/failoverMultipleNodesDialogBoxesChecked.png b/content/clustersetup/picts/failoverMultipleNodesDialogBoxesChecked.png new file mode 100644 index 0000000000..121ce448a0 Binary files /dev/null and b/content/clustersetup/picts/failoverMultipleNodesDialogBoxesChecked.png differ diff --git a/content/clustersetup/picts/failoverMultipleNodesFailoverDone.png b/content/clustersetup/picts/failoverMultipleNodesFailoverDone.png new file mode 100644 index 0000000000..9e0cce8c3d Binary files /dev/null and b/content/clustersetup/picts/failoverMultipleNodesFailoverDone.png differ diff --git a/content/clustersetup/picts/failoverMultipleNodesRebalanceDone.png b/content/clustersetup/picts/failoverMultipleNodesRebalanceDone.png new file mode 100644 index 0000000000..515f62ab8e Binary files /dev/null and b/content/clustersetup/picts/failoverMultipleNodesRebalanceDone.png differ diff --git a/content/clustersetup/picts/fullRecoveryPending.png b/content/clustersetup/picts/fullRecoveryPending.png new file mode 100644 index 0000000000..2c4ea93556 Binary files /dev/null and b/content/clustersetup/picts/fullRecoveryPending.png differ diff --git a/content/clustersetup/picts/groupsAccessMoveServerTab.png b/content/clustersetup/picts/groupsAccessMoveServerTab.png new file mode 100644 index 0000000000..970f62fc6c Binary files /dev/null and b/content/clustersetup/picts/groupsAccessMoveServerTab.png differ diff --git a/content/clustersetup/picts/groupsAddGroupDialog.png b/content/clustersetup/picts/groupsAddGroupDialog.png new file mode 100644 index 0000000000..65f1f6c05a Binary files /dev/null and b/content/clustersetup/picts/groupsAddGroupDialog.png differ diff --git a/content/clustersetup/picts/groupsAddGroupTab.png b/content/clustersetup/picts/groupsAddGroupTab.png new file mode 100644 index 0000000000..c34bd1612c Binary files /dev/null and b/content/clustersetup/picts/groupsAddGroupTab.png differ diff --git a/content/clustersetup/picts/groupsApplyChangesButton.png b/content/clustersetup/picts/groupsApplyChangesButton.png new file mode 100644 index 0000000000..725de8428c Binary files /dev/null and b/content/clustersetup/picts/groupsApplyChangesButton.png differ diff --git a/content/clustersetup/picts/groupsConfirmServerRemovalNotice.png b/content/clustersetup/picts/groupsConfirmServerRemovalNotice.png new file mode 100644 index 0000000000..884fac8bc8 Binary files /dev/null and b/content/clustersetup/picts/groupsConfirmServerRemovalNotice.png differ diff --git a/content/clustersetup/picts/groupsDeleteGroup.png b/content/clustersetup/picts/groupsDeleteGroup.png new file mode 100644 index 0000000000..2cbe1c4720 Binary files /dev/null and b/content/clustersetup/picts/groupsDeleteGroup.png differ diff --git a/content/clustersetup/picts/groupsDeleteGroupConfirmation.png b/content/clustersetup/picts/groupsDeleteGroupConfirmation.png new file mode 100644 index 0000000000..bdef2ce849 Binary files /dev/null and b/content/clustersetup/picts/groupsDeleteGroupConfirmation.png differ diff --git a/content/clustersetup/picts/groupsEditGroupNameDialog.png b/content/clustersetup/picts/groupsEditGroupNameDialog.png new file mode 100644 index 0000000000..07f2687d78 Binary files /dev/null and b/content/clustersetup/picts/groupsEditGroupNameDialog.png differ diff --git a/content/clustersetup/picts/groupsEditGroupNameTab.png b/content/clustersetup/picts/groupsEditGroupNameTab.png new file mode 100644 index 0000000000..c05f2907ee Binary files /dev/null and b/content/clustersetup/picts/groupsEditGroupNameTab.png differ diff --git a/content/clustersetup/picts/groupsEmptyGroup.png b/content/clustersetup/picts/groupsEmptyGroup.png new file mode 100644 index 0000000000..6f25bc4609 Binary files /dev/null and b/content/clustersetup/picts/groupsEmptyGroup.png differ diff --git a/content/clustersetup/picts/groupsFollowingServerRemoval.png b/content/clustersetup/picts/groupsFollowingServerRemoval.png new file mode 100644 index 0000000000..c4ca402f0e Binary files /dev/null and b/content/clustersetup/picts/groupsFollowingServerRemoval.png differ diff --git a/content/clustersetup/picts/groupsPendingMoveNotification.png b/content/clustersetup/picts/groupsPendingMoveNotification.png new file mode 100644 index 0000000000..ae228d05cc Binary files /dev/null and b/content/clustersetup/picts/groupsPendingMoveNotification.png differ diff --git a/content/clustersetup/picts/groupsRebalanceButton.png b/content/clustersetup/picts/groupsRebalanceButton.png new file mode 100644 index 0000000000..bdedf04fa2 Binary files /dev/null and b/content/clustersetup/picts/groupsRebalanceButton.png differ diff --git a/content/clustersetup/picts/groupsRebalanceFollowingMove.png b/content/clustersetup/picts/groupsRebalanceFollowingMove.png new file mode 100644 index 0000000000..2dcfffc8ca Binary files /dev/null and b/content/clustersetup/picts/groupsRebalanceFollowingMove.png differ diff --git a/content/clustersetup/picts/groupsRemovalPendingNotification.png b/content/clustersetup/picts/groupsRemovalPendingNotification.png new file mode 100644 index 0000000000..ca1f1a37cf Binary files /dev/null and b/content/clustersetup/picts/groupsRemovalPendingNotification.png differ diff --git a/content/clustersetup/picts/groupsResetAndApplyChangesButtons.png b/content/clustersetup/picts/groupsResetAndApplyChangesButtons.png new file mode 100644 index 0000000000..b052b21342 Binary files /dev/null and b/content/clustersetup/picts/groupsResetAndApplyChangesButtons.png differ diff --git a/content/clustersetup/picts/groupsScreenFollowingMoveServer.png b/content/clustersetup/picts/groupsScreenFollowingMoveServer.png new file mode 100644 index 0000000000..ef6c265435 Binary files /dev/null and b/content/clustersetup/picts/groupsScreenFollowingMoveServer.png differ diff --git a/content/clustersetup/picts/groupsScreenWithOpenGroup.png b/content/clustersetup/picts/groupsScreenWithOpenGroup.png new file mode 100644 index 0000000000..5fa0fc6c9f Binary files /dev/null and b/content/clustersetup/picts/groupsScreenWithOpenGroup.png differ diff --git a/content/clustersetup/picts/groupsScreenWithTwoGroups.png b/content/clustersetup/picts/groupsScreenWithTwoGroups.png new file mode 100644 index 0000000000..ce28fd1cdf Binary files /dev/null and b/content/clustersetup/picts/groupsScreenWithTwoGroups.png differ diff --git a/content/clustersetup/picts/groupsSelectFromAddServerDialog.png b/content/clustersetup/picts/groupsSelectFromAddServerDialog.png new file mode 100644 index 0000000000..ee1beaccec Binary files /dev/null and b/content/clustersetup/picts/groupsSelectFromAddServerDialog.png differ diff --git a/content/clustersetup/picts/groupsSelectGroupsTab.png b/content/clustersetup/picts/groupsSelectGroupsTab.png new file mode 100644 index 0000000000..3ff6da5477 Binary files /dev/null and b/content/clustersetup/picts/groupsSelectGroupsTab.png differ diff --git a/content/clustersetup/picts/groupsServerGroupsNewGroupAdded.png b/content/clustersetup/picts/groupsServerGroupsNewGroupAdded.png new file mode 100644 index 0000000000..3f065e7b48 Binary files /dev/null and b/content/clustersetup/picts/groupsServerGroupsNewGroupAdded.png differ diff --git a/content/clustersetup/picts/groupsServerGroupsScreen.png b/content/clustersetup/picts/groupsServerGroupsScreen.png new file mode 100644 index 0000000000..25392f5d8a Binary files /dev/null and b/content/clustersetup/picts/groupsServerGroupsScreen.png differ diff --git a/content/clustersetup/picts/groupsServersUI.png b/content/clustersetup/picts/groupsServersUI.png new file mode 100644 index 0000000000..b2043ba6b2 Binary files /dev/null and b/content/clustersetup/picts/groupsServersUI.png differ diff --git a/content/clustersetup/picts/groupsTab.png b/content/clustersetup/picts/groupsTab.png new file mode 100644 index 0000000000..77931f0da9 Binary files /dev/null and b/content/clustersetup/picts/groupsTab.png differ diff --git a/content/clustersetup/picts/loggingScreenBasic.png b/content/clustersetup/picts/loggingScreenBasic.png new file mode 100644 index 0000000000..4474dbe580 Binary files /dev/null and b/content/clustersetup/picts/loggingScreenBasic.png differ diff --git a/content/clustersetup/picts/logredaction.png b/content/clustersetup/picts/logredaction.png new file mode 100644 index 0000000000..b01bfd01bb Binary files /dev/null and b/content/clustersetup/picts/logredaction.png differ diff --git a/content/clustersetup/picts/newBucketDisplay.png b/content/clustersetup/picts/newBucketDisplay.png index da4f19485b..9f841923d4 100644 Binary files a/content/clustersetup/picts/newBucketDisplay.png and b/content/clustersetup/picts/newBucketDisplay.png differ diff --git a/content/clustersetup/picts/partialRedactionSelection.png b/content/clustersetup/picts/partialRedactionSelection.png new file mode 100644 index 0000000000..660456f952 Binary files /dev/null and b/content/clustersetup/picts/partialRedactionSelection.png differ diff --git a/content/clustersetup/picts/recoveryOptions.png b/content/clustersetup/picts/recoveryOptions.png new file mode 100644 index 0000000000..95a81cd122 Binary files /dev/null and b/content/clustersetup/picts/recoveryOptions.png differ diff --git a/content/clustersetup/picts/uploadToCouchbaseCheckbox.png b/content/clustersetup/picts/uploadToCouchbaseCheckbox.png new file mode 100644 index 0000000000..406682e314 Binary files /dev/null and b/content/clustersetup/picts/uploadToCouchbaseCheckbox.png differ diff --git a/content/clustersetup/picts/uploadToCouchbaseDialogBasic.png b/content/clustersetup/picts/uploadToCouchbaseDialogBasic.png new file mode 100644 index 0000000000..15e9251669 Binary files /dev/null and b/content/clustersetup/picts/uploadToCouchbaseDialogBasic.png differ diff --git a/content/clustersetup/rebalance.dita b/content/clustersetup/rebalance.dita index 044891d13e..bd1a099870 100644 --- a/content/clustersetup/rebalance.dita +++ b/content/clustersetup/rebalance.dita @@ -20,8 +20,8 @@
    Rebalancing the Data Service

    As explained in , data is - logically partitioned between all of the nodes running the Data service in the cluster. When + href="../understanding-couchbase/clusters-and-availability/clusters-and-availability.dita"/>, data is + logically partitioned between all of the nodes running the Data Service in the cluster. When these nodes are rebalanced, the vBuckets are redistributed evenly between all of the Data service nodes in the cluster. In the case of nodes being removed from the cluster, each vBucket from the removed node is replicated to the new active node for that vBucket, which @@ -71,8 +71,11 @@

    Rebalancing Multiple Services -

    Usually Couchbase Server deployments have a single service deployed on each node, see . In instances where +

    Frequently, Couchbase Server production-deployments have no + more than one service deployed on each node, see + . + + In instances where this is not the case, the rebalance behavior will be the combination of all services deployed on the node.

    diff --git a/content/clustersetup/rejoin-cluster.dita b/content/clustersetup/rejoin-cluster.dita index 1066a8c7e3..a0e5f681e7 100644 --- a/content/clustersetup/rejoin-cluster.dita +++ b/content/clustersetup/rejoin-cluster.dita @@ -1,154 +1,344 @@ - Rejoining a Cluster - From the Couchbase Web Console, the Delta Recovery and Full Recovery options - display after the server node is failed over. + + Recovering a Node + + + + After a node has been failed over, it can be recovered: that is, added back into + the cluster from which it was failed over, by means of rebalancing. + + -

    Both recovery methods add the server node back into the cluster during the rebalance - operation. However, full recovery removes the node's data prior to the rebalance and delta - recovery schedules the node's existing data to be reused.

    -
    - Full Recovery - -

    Full recovery characteristics are as follows:

    + +
    + + Performing Node Recovery + + +

    + After a node has been failed over, there are two principal options: +

    +
      -
    • Data files are removed from the server node.
      • -
    • Indexes must be rebuilt on the server node that is being re-added.
      • -
    • A working set of documents is restored for the vBuckets that are being moved.
      • -
    • It is used when the data size is smaller than the bucket quotas. In this case, moving - data over the network and on the disk (full recovery) may be faster than warming up data - files (delta recovery).
      • +
    • + The node can be removed from the cluster, by means of the + rebalance operation. + +

      + +

      + +
      • + +
    • + The node can be recovered, and thereby added back into the cluster — again by means of + the rebalance operation. + +

      + +

      +
      • +
    + +

    + These options are displayed by Couchbase Web Console as follows: +

    + +

    + +

    + +

    + Removal of a failed-over node is appropriate when you have no intention or ability to recover the failed-over node: + for example, when full hardware-replacement is required. Since the REMOVAL pending rebalance notification + is now displayed, to remove the node, + simply left-click on the Rebalance button, at the upper-right. +

    + +

    + Alternatively, Recovery of the node may be more appropriate: for example, when the node's availability has been restored, + and no other key aspect of the node's integrity has been impaired. + To recover a failed-over node, you must select a recovery option, before performing a rebalance. Two recovery options + are provided, by the buttons at the lower right; which are Add Back: Full Recovery and + Add Back: Delta Recovery. After you have selected the appropriate option, you then Rebalance. +

    + +

    + For example, if you left-click on Add Back: Full Recovery, the display changes + as follows: +

    + +

    + +

    + +

    + This now indicates that a Full Recovery will occur when you left-click the Rebalance button. +

    + +

    + The Full and Delta recovery-options are described below. +

    +
    -
    Delta Node Recovery -

    Delta node recovery permits nodes to be re-added to the cluster - and incrementally caught up with the data changes.

    -

    Delta node recovery permits a failed over node to be recovered by re-using the data on its - disk and then resynchronizing the data based on the delta change. - The failed over node is checked to identify the point when data mutations stopped and resynchronized beginning at that point. - The server node catches up on data mutations for its vBuckets and starts serving data. - Because the original data and data buckets are retained, the cluster starts functioning with minimal downtime. - - Index nodes retain their index definitions when being failed over and recovered using delta recovery, although the data - within the indexes is deleted. - When they are re-added to the cluster, the indexes will be automatically rebuilt by the Index service. + + +

    - This operation improves recovery time and network resource usage. - - Server nodes are removed from clusters under many circumstances. - The following are circumstances (among many others) where a server node might be re-added to the cluster after being failed over. + + Full Recovery + + +

    + Full recovery involves removing all pre-existing data from, and assigning new + data to, the node that is being recovered. + Therefore, when this option has been specified, left-clicking on Rebalance causes + the following:

    +
      -
    • Node goes down for a short period of time
      • -
    • Routine maintenance is scheduled
      • -
    • Network connectivity is briefly disrupted
      • +
    • + All existing vBuckets and documents are removed from the node. + +

      +

      + +
      • + +
    • + If GSI Indexes reside on the node, they are left unmodified during the rebalance process. + +

      + +

      + +
      • + +
    • + A new set of vBuckets and documents is assigned to the node. + +

      + +

      + +
      • + +
    • + When the node's vBuckets are all up to date, and the rebalance process concludes, the node recommences the serving of data. + If GSI Indexes reside on the node, they become active, and are updated by the Index Service as appropriate. + +

      + +

      + +
      • +
    -

    When a node is failed over, data files are preserved. - The data files are used either for Couchbase support, data recovery, or delta node recovery.

    -

    In the process of failing over a node, performing maintenance, adding the node back into the cluster, - and rebalancing, data is recovered via either full recovery mode or delta recovery mode. - With delta recovery mode, Couchbase detects (with the Database Change Protocol) - which data files are up-to-date and which are out-of-date and then, during rebalance, - the existing data files on the failed over server node are retained and the out-of-date files are updated.

    -

    From the web console, the Delta Recovery and Full Recovery options display after the server node is failed over. - Both recovery methods add the server node back into the cluster during the rebalance operation, - however, full recovery removes the node's data prior to the rebalance and delta recovery - schedules the node's existing data to be re-used.

    -

    Delta recovery requirements:

    + +
    + +
    + + + Delta Recovery + + +

    + Delta recovery maintains and resynchronizes a node's pre-existing data. Therefore, when this option + has been specified, left-clicking on Rebalance causes + the following: +

    +
      -
    • A healthy server node and a healthy state for the cluster.
      • -
    • The server node is failed over. Delta recovery is not possible for - a rebalance-in operation (add server) or rebalance-out operation (remove server).
      • -
    • Delta recovery must be possible for all buckets. - For example, if delta recovery is possible for a subset of buckets but not possible for another - subset of buckets, then the Couchbase cluster does not permit a rebalance operation.
      • -
    • Because delta recovery relies on the existing data files on the failed over server node's disk, - the exact same set of buckets must be transferred to the failed over server node.
      • +
    • + No existing vBucket or document is removed from the node. +

      + +

      +
      • + +
    • + All existing data is loaded into memory. +

      + +

      +
      • + +
    • + The point at which mutations to node-data most recently stopped is determined. Then, vBuckets are duly + updated from that point; based on the data-changes that have since occurred, elsewhere on the cluster. + +

      + +

      +
      • + +
    • + If GSI Indexes reside on the node, they are left unmodified during the rebalance process. +

      + +

      +
      • + +
    • + When the node's vBuckets are all up to date, and the rebalance process concludes, the node recommences the serving of data. If + GSI Indexes reside on the node, they become active, and are updated by the Index Service as appropriate (this includes + updates that correspond to whatever mutations were made by the Data Service while the node was in a failed over state). +

      + +

      +
      • +
    -

    Delta recovery characteristics:

    + +
    + +
    + + + Delta-Recovery Requirements + + +

    + Delta recovery can only be performed when: +

    +
      -
    • Data files are "warmed up" into memory. Warmed up into memory means that data is loaded into memory. - As a minimum, depending on whether metadata is retained in or not, all data file keys are loaded - from disk prior to the rebalance operation.
      • -
    • Index definitions are retained on the node being recovered. Once the node has been re-added, these - indexes are automatically rebuilt by the Index service.
      • -
    - - An environment with a large data footprint might use delta node recovery when re-adding a failed over server node. -

    Failure Scenarios

    -

    There are conditions where delta node recovery either defaults to full recovery or is not available.

    -

    The following are conditions for delta node recovery failures:

    +
  • + The node to be recovered is healthy; likewise the cluster. +

    + +

    +
    • + +
  • + The node to be recovered is in a failed over state. +

    + +

    +
    • + +
  • + Delta recovery is possible for all Buckets on the node. Note that the correspondence of + Buckets, vBuckets, and documents on the node remains entirely unchanged by Delta recovery. +

    + +

    +
    • + +

+ + + +
+ + + Delta-Recovery Failures + + +

+ In some cases, when Delta recovery is attempted, and all the requirements listed immediately above have + been met, the operation nevertheless either fails or defaults to Full + recovery. This indicates that one or more of the following conditions exist: +

+
    -
  • If topology changes occur while a node is pending delta recovery, delta node recovery is impacted. - For example, another node is added, a node is removed, or a node is swapped.
    • -
  • If a down node is hard failed over and is marked for removal.
    • -
  • If rebalance-in-out operations are performed where the number of in and out nodes do not match (swap rebalance works in this case).
    • -
  • If certain bucket operations are performed while a node is pending delta recovery, delta node recovery is impacted. - For example, a new bucket is added, a bucket's replica configuration is changed, or a bucket is flushed.
    • +
  • + Cluster-topology has changed since the node was last available within the cluster. +

    + +

    +
    • + +
  • + The node was hard failed over, and is marked for removal. +

    + +

    +
    • + +
  • + The rebalance was configured to perform the Delta recovery while simultaneously moving other nodes in or + out of the cluster, and the numbers of nodes intended respectively to leave and join the + cluster were unequal. (Note that in this case, a Swap Rebalance can be performed instead.) +

    + +

    +
    • + +
  • + Bucket-operations were performed while Delta recovery was pending: this changed configurations, and has made + Delta recovery impossible. +

    + +

    +
    • + +
  • + Either the node being recovered or another node in the cluster has crashed, or become otherwise unavailable, at some + point during the process of recovery and rebalance. +

    + +

    +
    • +
-

The following describes scenarios where delta node recovery either defaults to full recovery or is not available.

-
- -
Node 1 is in delta recovery and Node 2, an active server node, crashes.
-
-
    -
  1. Node 1 is failed over and delta recovery is specified. Now, Node 1 is pending delta recovery.
    2. -
  2. Node 2, an active server, goes down. - The rebalance operation is not available.
    3. -
  3. Fail over Node 2.
    4. -
  4. Cancel the pending delta recovery, specify full recovery, and rebalance.
    5. -
  5. Repair Node 2, add the server to the cluster, and rebalance.
    6. -
-
- -
-
- -
Node 1 is in delta recovery and Node 1 crashes during rebalance.
-
-
    -
  1. Node 1 is failed over, delta recovery is specified, and the rebalance operation is started.
    2. -
  2. Node 1 crashes and the rebalance operation fails.
    3. -
  3. Repair Node 1, re-start the server node, and rebalance. Node1 is added back to the cluster using full recovery.
    4. -
- -
- -
-
- -
Node 1 is in delta recovery and a bucket operation is performed.
-
The bucket operations that cause rebalance to fail are adding bucket, changing replica configuration, or flushing bucket
-
-
    -
  1. Node 1 is failed over, delta recovery is specified, and then a bucket operation is performed.
    2. -
  2. Rebalance is performed and fails.

    3. -
  3. Cancel the pending delta recovery, specify full recovery, and rebalance. - Bucket deletion does not lead to delta recovery failure. -
    4. -
- -
- -
- -
- -
Node 1 and Node 2 are in delta node recovery and Node 2 crashes.
-
-
    -
  1. Both Node 1 and Node 2 are failed over, delta recovery is specified.
    2. -
  2. Node 2 crashes.
    3. -
  3. Rebalance is performed and fails.

    4. -
- -
- -
+ +
+ +
+ + + Recovery Performance + + +

+ In many cases, Delta recovery is faster than Full recovery; since a significant quantity of + usable data already resides on the node, and therefore does not require network-transfer: only updates made + since the node's last-recorded mutation need to be accessed from other nodes in the cluster. +

+ +

+ However, in cases where the node's memory-footprint is extremely large, and data-size exceeds bucket memory-quotas, + the memory-management overhead potentially entailed by Delta recovery might imply + Full recovery's taking less time overall. +

+ +
+ +
+ + + Performing Recovery with CLI + + +

+ To select a recovery option with the Couchbase CLI, see the + recovery command. To + perform the subsequent rebalance, see the + rebalance command. +

+ +
+ +
+ + + Performing Recovery with REST + + +

+ To select a recovery option with the Couchbase REST API, see + Setting Recovery Type. To + perform the subsequent rebalance, see + Rebalancing Nodes +

+
+ diff --git a/content/clustersetup/remove-nodes.dita b/content/clustersetup/remove-nodes.dita index 9661fe0c8c..5057d5afe6 100644 --- a/content/clustersetup/remove-nodes.dita +++ b/content/clustersetup/remove-nodes.dita @@ -16,7 +16,7 @@ available for queries and are dropped. -
  • Query service - loss of query capability in that node.
    • +
  • Query, Search, Analytics or Eventing service - loss of service-capability on that node.

    • Before you remove a node from the cluster, ensure that you have the capacity within the remaining nodes to handle the workload. Removing a node does not stop the node from servicing @@ -81,6 +81,6 @@ - + diff --git a/content/clustersetup/server-setup.dita b/content/clustersetup/server-setup.dita index 9c6c044173..d0bdca3978 100644 --- a/content/clustersetup/server-setup.dita +++ b/content/clustersetup/server-setup.dita @@ -146,7 +146,7 @@ Sub-document path length 1024 bytes The maximum length of a sub-document path that may be used in one - of the sub-document operations. diff --git a/content/clustersetup/ui-logs.dita b/content/clustersetup/ui-logs.dita index 7fcb304c4f..a24d0e294c 100644 --- a/content/clustersetup/ui-logs.dita +++ b/content/clustersetup/ui-logs.dita @@ -223,6 +223,40 @@ + +

    + <b><b>Log Redaction</b></b> +

    You can use logs for multiple purposes ranging from security, monitoring, and diagnostics. + Suppression of sensitive data such as personally identifiable information (PII), hostnames, + internal asset information, credit card details, during the logging operation is termed as + log redaction. Organizations implement log redaction as part of their legal compliance and + security risk mitigations. Couchbase Server provides a capability to redact sensitive user + data from getting captured in the logs. All sensitive data are scrubbed and gets removed + from the log files. Post redaction, log files can be shared for troubleshooting without + disregarding any regulatory compliance.

    +

    To enable log redaction:

      +
    1. From the Couchbase Web Console > Logs page > select the Collect + Information.

      2. +
    2. Select all nodes or use the filter nodes… option to select nodes from where you + want to collect logs and diagnostic information.
      3. +
    3. From the Redact Logs pane select No Redaction or Partial + Redaction option.
        +
      • No Redaction: Select this option to enable capturing of log data that may + include any sensitive information.
        • +
      • Partial Redaction. Select this option to enable abstraction of sensitive + information in the log files.

        When the Partial Redaction option is selected, + Couchbase automatically stores two types of files, one with redaction and another + without data redaction. Use the log file without data redaction for internal + troubleshooting purposes, and in case you are using the Upload to Couchbase + log option, use the log file with data redaction. The above information + message also gets displayed in the Web console UI.

        UI Message:"Couchbase + Server will collect and save a redacted log file at the location you specify, but + also save an unredacted version which could be useful for further troubleshooting. + If you use the "Upload to Couchbase" feature below, ONLY the redacted log will be + uploaded."

        • +
      4. +

    Manage Logs @@ -281,7 +315,7 @@ cluster, views, mapreduce_errors , xdcr and error_logger.
  • logging_level - The available log levels are - debug, info, warning, and + debug, info, warn, and error.curl -X POST -u Administrator:password http://127.0.0.1:8091/diag/eval \ -d 'ale:set_loglevel(ns_server,error).

    • @@ -295,9 +329,9 @@ one of the most important diagnostic tools used by Couchbase technical support.

    The other three CLI commands you can use to start and stop log collection and to read the log collection status:

      -
      • -
      • -
      • +
      • +
      • +

    diff --git a/content/concepts/concepts-architecture-intro.dita b/content/concepts/concepts-architecture-intro.dita index decba2d1a1..e44adeaf4b 100644 --- a/content/concepts/concepts-architecture-intro.dita +++ b/content/concepts/concepts-architecture-intro.dita @@ -4,13 +4,6 @@ Concepts and architecture The Concepts and architecture guide introduces the key concepts in Couchbase Server and provides information about the Couchbase Server architecture. -

    This guide contains the following topics:

      -

    • Introduces the key concepts in Couchbase - Server.

      • -

    • Discusses the data modeling exercise and evolving data models in Couchbase.

      • -

    • Introduces N1QL and provides a comparison between traditional SQL and N1QL.

      • -

    • Provides detailed information about the Couchbase Server architecture.

      • -
    -

    + diff --git a/content/concepts/indexing.dita b/content/concepts/indexing.dita index e5b5818715..0d289a882d 100644 --- a/content/concepts/indexing.dita +++ b/content/concepts/indexing.dita @@ -52,7 +52,8 @@
  • Full Text Indexes allow developers to easily add full-text search capabilities to their application using CBFT (developer preview), search use cases without deploying additional components, which reduces operational complexity.
    • -

    For more information on how the indexers and index services work, see .

    +

    For more information on how the indexers and index services work, see + .

     -

    - - -
  • - The node.js routine performs an Elasticsearch query on the existing beer-sample index: the returned - documents each contain an ID corresponding to a particular beer, which is described by the specified - style. - -

    - -

    -
    • - -
  • - The documents are passed back to the client-side, where the IDs are retrieved. Each is displayed for - the user. - -

    - -

    -
    • - -
  • - The client passes each ID to the node.js program cbNodeJsQueryAgent. - -

    - -

    -
    • - -
  • - The program cbNodeJsQueryAgent duly queries Couchbase. Couchbase returns, for - each ID, a document containing detailed information on the beer specified by the ID. - -

    - -

    -
    • - -
  • - Each document is returned to the client-side routine, which displays the results for - the user. - -

    - -

    -
    • - - -
    - -
    - - Access Source-Files - - -

    - You can access the three source-files for the example at this location: - . The - following sections of the current document provide a brief summary of the functionality. -

    -
    - -
    - - Client-Side HTML and JavaScript - - -

    - The file couchbaseESqueryDemo.html provides html-based interactive elements for the selection of beer-styles and the - display of query-results. Beer-styles can be selected by means of a series of radio-buttons, within a dialog named - availableBeerStylesDialog. -

    - -

    - A value is associated with each possible radio-button selection. When the user left-clicks on the Query - Elasticsearch button, this value is retrieved: -

    - - $("#queryElasticsearch").click(function(event) -{ - // Get the user's radio-button selection, which corresonds to a particular - // beer style. - // - var beerStyles = document.getElementsByName("beerStyle"); - var selectedBeerValue = 0; - var selectedBeerStyle = ""; - - for (var i = 0; i < beerStyles.length; i++) - { - if (beerStyles[i].checked == true) - { - selectedBeerValue = beerStyles[i].value; - } - } - -

    - The value is then used to determine the style-name, which will be passed to Elasticsearch, and used in - a search-procedure. -

    - - if (selectedBeerValue == 0) -{ - selectedBeerStyle = "American-Style Pale Ale"; -} -else -{ - if (selectedBeerValue == 1) - { - selectedBeerStyle = "American-Style Brown Ale"; - } - else - { - if (selectedBeerValue == 2) - -

    - A corresponding Elasticsearch query is then prepared and executed: -

    - - var esNodeJsAddress = "http://localhost:8081/"; -var esNodeJsTargetURL = esNodeJsAddress + '?' + "foo=" + selectedBeerStyle; - -$.getJSON(esNodeJsTargetURL, function(dataFromElasticsearch) -{ - -

    - Retrieved IDs are, first, assembled and displayed: -

    - - $.each(dataFromElasticsearch, function(key, val) -{ - esDataDisplayString = esDataDisplayString + '<p>' + dataFromElasticsearch[key]._id + '</p>'; - numberOfIds++; -}); - -document.getElementById('ElasticSearchRetrievalsContent').innerHTML = esDataDisplayString; - -

    - Then, the IDs are dispatched as an array, to cbNodeJsQueryAgent, so that Couchbase Server - can be searched for each of them. -

    - - cbNodeJsTargetURL = cbNodeJsAddress + '?' + keyNameForIDparam + '=' -+ returnedCouchbaseIDsStringed + '&' + countKey + '=' + numberOfIds; - -$.getJSON(cbNodeJsTargetURL, function(dataReturnedFromCouchbase) -{ - -

    - An array of documents is returned, each of which is duly displayed: -

    - - for (var currentKeyPosition = 0; currentKeyPosition < numberOfIds; currentKeyPosition++) -{ - - $.each(JSON.parse(dataReturnedFromCouchbase[currentKeyPosition]), function(key, val) - { - cbDataDisplayString = '<p>' + cbDataDisplayString + "\"" + key + "\"" - + " : " + "\"" + val + "\"" + '</p>'; - }); -} - -document.getElementById('CouchbaseRetrievalsContent').innerHTML = cbDataDisplayString; - -
    - -
    - - Server-Side node.js for Elasticsearch - - -

    - The file esNodeJsQueryAgent.js uses the require function to add appropriate modules, including - the module for the Elasticsearch client. It then creates an instance of the client: -

    - - var http = require('http'); -var url = require('url'); -var elasticsearch = require('elasticsearch'); -var client = new elasticsearch.Client({ - host: 'localhost:9200', - log: 'trace' -}); - -

    - An http server is then created: -

    - - http.createServer(function (request, response) -{ -console.log('New connection'); - -

    - The server is directed (near the end of the file) to listen on port 8081: -

    - - }).listen(8081); - -

    - The value passed to the program by couchbaseESqueryDemo is retrieved, by referencing its known key: -

    - - var queryObject = url.parse(request.url, true).query - -var luceneString = queryObject.foo; - -

    - Then, an appropriate query-string is created, to be passed to - Elasticsearch: -

    - - var clientSearchStringStart = - "{index: 'beer-sample', body: { query: { query_string: { query: "; -var clientSearchStringEnd = "}}}}" -var clientSearchStringFull = clientSearchStringStart + "\"" + luceneString + "\"" + clientSearchStringEnd; -var extendedLuceneString = "\"" + luceneString + "\""; - -

    - Next, the Elasticsearch query is made, on the beer-sample index: -

    - - client.search({ - index:'beer-sample', - body: { - query: { - query_string: { - query: luceneString - } - } - } - }).then(function(resp){ - hits = resp.hits.hits; - console.log("Hits are: " + JSON.stringify(hits)); - -

    - The retrieved data is then passed back to couchbaseESqueryDemo: -

    - - response.writeHead(200, {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}); -response.end(JSON.stringify(hits)); - -
    - -
    - - Server-Side node.js for Couchbase - - -

    - Having used the required function to include modules for url and http, cbNodeJsQueryAgent creates an - http server, and specifies that it will listen on port 8080. -

    - -

    - It then parses the query-URL, and obtains the ID-array, provided by couchbaseESqueryDemo. It then - prepares to access Couchbase on its default port: -

    - - var queryObject = url.parse(request.url, true).query - -var couchbase = require("couchbase"); -var myCluster = new couchbase.Cluster('couchbase://localhost'); - -// Authentication, required by 5.0 and later... -// -myCluster.authenticate('beer-sample', 'beer-sample'); - -var myBucket = myCluster.openBucket('beer-sample'); - -

    - Note, in the code-fragment - immediately above, the line used for authentication, which is required by Couchbase - Server 5.0 and later. This particular example assumes that a user has been - defined whose username and password are both beer-sample. It - also assumes that the user has been assigned the Full Bucket Access role - for the beer-sample bucket. If a pre-5.0 version of Couchbase Server is - being used, this line can be commented out or deleted. See - Authorization, - for information on accessing server-resources by means of Role-Based Access - Control. -

    - -

    - Next, the function searchCouchbaseForNextID is called recursively, once - for each ID in the array. The function itself invokes the Couchbase SDK get method, - to search Couchbase Server for a single ID. (Note that this method is asynchronous.) Once - all IDs have been searched for, a response containing an array of retrieved documents - is provided to the client. -

    - - function searchCouchbaseForNextID(arrayOfIDs, count, totalCount) -{ - myBucket.get(arrayOfIDs[count], function(err, res) - { - couchbaseObjectArray[count] = JSON.stringify(res.value); - - if (count < totalCount) - { - count++; - searchCouchbaseForNextID(receivedArray, count, totalCount); - } - else - { - response.writeHead(200, {"Content-Type": "application/json", - "Access-Control-Allow-Origin": "*"}); - response.end(JSON.stringify(couchbaseObjectArray)); - } - }); -} - -
    - -
    - - Setting Up the Example - - -

    - Successful running of the example requires that Couchbase Server and Elasticsearch both be already - installed, configured, and running. The instructions in this section assume that all are on the same - node, and that all services can thus be accessed from localhost. -

    - -

    - Note that the node.js program esNodeJsQueryAgent.js has been written to run on port 8081; and - cbNodeJsQueryAgent.js on port 8080. If you wish to change these port-designations, you must edit the - program-files, including that for couchbaseESqueryDemo.html. -

    - -

    - To run the provided node.js programs, you must install both the Couchbase SDK and node.js - Elasticsearch client; which in turn requires that you install and use the Node Package Manager, npm. - On non-Windows platforms, you may also need to install node-gyp. -

    - -

    - For information on installing the node.js instance of the Couchbase SDK see the documentation at - Start - Using the SDK. - See also the Elasticsearch documentation for installing the Elasticsearch client, at - About. -

    - -

    - When you have installed the required SDK, start cbNodeJsQueryAgent at the command-line, as follows: -

    - - $ node cbNodeJsQueryAgent.js - -

    - The message Server started is provided in response. -

    - -

    - Start esNodeJsQueryAgent in a separate terminal, as follows: -

    - - $ node esNodeJsQueryAgent.js - -

    - A repsonse is provided, confirming that the program has been added as a connection to - http://localhost:9200, which is the Elasticsearch port. -

    - -

    - Now, bring up couchbaseESqueryDemo.html in a browser. The layout appears as follows: -

    - -

    - -

    - -

    - The UI features three principal elements. at the upper-left, a dialog presents a series of - radio-butons, permitting selection from a number of beer-styles. At the lower-left, - a pane is provided for the display of IDs retrieved from Elasticsearch; at the right, - a pane for the display of documents retrieved from Couchbase. -

    - -

    - Each beer-style can be selected by its corresponding radio-button. For example: - -

    - -

    - -

    - -

    - Once a beer-style has been selected, querying can be initiated by left-clicking on the - Query Elasticsearch button. -

    - -

    - -

    - -

    - The full query-routine is duly performed: the beer-style is passed to Elasticsearch, and - queried on; ID-information is returned to the client; then, ID-information is queried - against Couchbase Server. Elements retrieved from both repositories are displayed in the - appropriate panes: -

    - -

    - -

    - -

    - The panes can be scrolled as needed, to reveal the full set of results. -

    - -
    - -
    - - Using Advanced Lucene Strings - - -

    - In the above example, since a single beer-style was used as the basis for an Elasticsearch query, the - Lucene string submitted was simply the style-name. Note that (as in the case of the curl command-line - example provided earlier) more complex queries can be submitted, with the full syntactical form of the - query simply submitted as the string. For example, the string American-Style Pale Ale might be replaced - by American-Style Pale Ale AND American-Style Brown Ale. -

    - -
    - - - - - - diff --git a/content/connectors/elasticsearch-2.2/getting-started.html b/content/connectors/elasticsearch-2.2/getting-started.html new file mode 100644 index 0000000000..f848c5b142 --- /dev/null +++ b/content/connectors/elasticsearch-2.2/getting-started.html @@ -0,0 +1,15 @@ + + + + Redirecting… + + + + +
    +

    Redirecting…

    + Click here if you are not redirected. + +
    + + diff --git a/content/connectors/elasticsearch-2.2/images/xdcrInitial.png b/content/connectors/elasticsearch-2.2/images/xdcrInitial.png deleted file mode 100644 index bf525db9b2..0000000000 Binary files a/content/connectors/elasticsearch-2.2/images/xdcrInitial.png and /dev/null differ diff --git a/content/connectors/elasticsearch-2.2/install-and-config.dita b/content/connectors/elasticsearch-2.2/install-and-config.dita deleted file mode 100644 index 2ef61c42f0..0000000000 --- a/content/connectors/elasticsearch-2.2/install-and-config.dita +++ /dev/null @@ -1,425 +0,0 @@ - - - - - - - Installation and Configuration - - - - A Couchbase-Elasticsearch data-replication system consists of three principal - components: a Couchbase Server-cluster of one or more nodes; an Elasticsearch - cluster of one or more nodes; and the Elasticsearch Transport Plug-in, installed - in the Elasticsearch environment. This section provides a step-by-step procedure, - whereby the three components can be installed, configured, and run. - - - - -
    - - - Installation Prerequisites - - -

    - The installation-procedure provided below involves the creation of a test environment, by means of Oracle VirtualBox; - requiring Ubuntu 14.04 or 16.04 as operating system. For details on setting up VirtualBox on your development-machine, and deploying - a Ubuntu image on it, see . - The instructions in this section assume that you have completed the necessary - procedures, and have supervisor-privileges in the resulting virtual environment. You will install and run Elasticsearch, - the Elasticsearch Plug-in, and Couchbase Server in this environment: you should thus ensure that the virtual machine - is provisioned with sufficient disk and memory resources; by means of the Settings panel, within the Oracle VM VirtualBox - Manager. -

    - -

    - Within the Ubuntu environment, you will use both the terminal and the browser to perform installation and configuration. - Note that within the terminal window, you may need to use supervisor-privileges, by means of the - sudo command: this command, - however, is not reproduced in the examples in this document; you must prepend your console-input with it, as appropriate. -

    - -
    - -
    - - - Install and Configure Couchbase Server - - -

    - Within the virtual environment you have created, using the browser, read and follow the installation and - configuration instructions provided at . - During configuration, note that for current test-purposes: -

    - -
      -
    • - You need only activate the Data service. Other services are not required. - -

      - -

      - -
      • - -
    • - You require a single bucket: which is the beer-sample bucket. This bucket will be used to provide data to - Elasticsearch, during the test. - -

      - -

      - -
      • - -
    • - If you are using Couchbase Server 5.0 or later, you should be familiar with the - Role-Based Access Control (RBAC) system now applied to all server-resources, - including buckets. Specifically, this - requires that the beer-sample bucket be accessed - by a defined user, with corresponding username and password. - This user must have been assigned a role that permits - access to the beer-sample bucket. See - Authorization - for information on Couchbase RBAC. See - Creating - and Managing Users with the UI for specific instructions on creating a user, - and on assigning role and credentials. - -

      - -

      -
      • -
    - -

    - Start Couchbase Server, using the instructions provided at . -

    -
    - -
    -Install Oracle Java - -

    - Oracle java (rather than the standard Ubuntu-available version) is required for both Elasticsearch and the - plug-in. Ensure Oracle Java is added to your system, by bringing up a terminal window, and proceeding as follows: -

    - -
      -
    1. Add the required Oracle Java repository: - - $ apt-add-repository ppa:webupd8team/java - -
      2. - -
    2. Install Oracle Java, and establish it as the default Java for the virtual machine: - - $ apt-get update -$ apt-get install oracle-java8-installer -$ apt install oracle-java8-set-default - -

      - During this process, when requested, accept the license-terms by selecting Ok or Yes. -

      - -

      - Once installation is complete, verify that Java has been successfully installed; by checking - its version, with java -version; and its location, with which java. -

      - -
      3. - -
    -
    - -
    - - Install Elasticsearch 2.4.0 - - -
      -
    1. Download the Elasticsearch 2.4.0 package for Ubuntu: - - $ wget https://download.elastic.co/elasticsearch/\ -> elasticsearch/elasticsearch-2.4.0.deb - -
      2. - -
    2. Install the package: - - $ dpkg -i elasticsearch-2.4.0.deb - -
      3. -
    - -
    - -
    - -Install the Elasticsearch Plug-in - -
      - -
    1. Navigate the the Elasticsearch install directory: - - $ cd /usr/share/elasticsearch - -
      2. - -
    2. Download and install the plug-in package: - - $ bin/plugin install https://github.com/couchbaselabs/\ -> elasticsearch-transport-couchbase/releases/download/2.2.4.0-update1/\ -> elasticsearch-transport-couchbase-2.2.4.0-update1.zip - -

      - Respond to the installation-prompts as appropriate. -

      - -

      - When installation is successful, the message Installed transport-couchbase - into /usr/share/elasticsearch/plugins/transport/couchbase is displayed. -

      - -
      3. - -
    -
    - -
    - - Configure and Start Elasticsearch - - -
      -
    1. - In preparation for configuration-file editing, install your preferred - console-based text-editor into the virtual machine environment. -

      - -

      - -

      - For example, download emacs, using the following command; then, - as installation proceeds, respond - to system-prompts as appropriate. - - $ apt-get install emacs -

      -
      2. - -
    2. - Using the editor, bring up the configuration file for Elasticsearch: - - emacs /etc/elasticsearch/elasticsearch.yml - -

      - Add the following to the end of the file, specifying an appropriate administrator-username and password - for accessing Elasticsearch: -

      - - couchbase.password: [password] -couchbase.username: [username] -couchbase.maxConcurrentRequests: 1024 - -

      - Then save the file, and exit back to the command-prompt. -

      - -
      3. - -
    3. - Install the Elasticsearch web-based user-interface: - - $ bin/plugin install mobz/elasticsearch-head - -
      4. - -
    4. - Start Elasticsearch: - - $ /etc/init.d/elasticsearch start - -
      5. - -
    5. - Configure an Elasticsearch index-template: - - $ curl -XPUT http://localhost:9200/_template/couchbase -d \ -> @plugins/transport-couchbase/couchbase_template.json - -

      - When successful, the configuration-routine provides the following response: - {"acknowledged":true}. -

      -
      6. - -
    6. - Instantiate an elasticsearch index to be applied by Elasticsearch to data from - a particular Couchbase bucket (in this case, beer-sample, which you - previously used in your set-up of Couchbase Server): - - $ curl -XPUT http://localhost:9200/beer-sample - -

      - When index-creation is successful, a further {"acknowledged":true} response - is provided. -

      -
      7. - -
    - -

    - You have now completed all basic installation and configuration requirements for - Elasticsearch, the Elasticsearch Plug-in, and Couchbase Server. (Note that more - advanced configuration options are described later in this document.) -

    - -
    - -
    - - Installing on Other Platforms - - -

    - The procedures and code-examples that you will encounter in the next section, - Getting Started, - all assume that you have installed and configured Elasticsearch, Couchbase Server, and - the Elasticsearch Plug-in precisely as described above. -

    - -

    - When you have completed Getting Started, and the other sections of this document, - and have thus become more familiar with the plug-in and its capability, you will likely - wish to perform installation on additional platforms, in preparation for or as part of - a production-deployment. -

    - -

    - Full details on the installation-procedures for Couchbase Server are described - here. - Those for Elasticsearch, - here. - Installing and configuring the plug-in on other platforms - is virtually the same as described here, for Ubuntu 14. Note that: -

    - -
      -
    • - The plug-in should be installed in the Elasticsearch install directory, with - bin/plugin install ...etc. - -

      - -

      -
      • - -
    • - Certain commands differ across platforms. For example, on CentOS, to install - the Elasticsearch package use yum (rather than dpkg, - as used on Ubuntu). - -

      - -

      -
      • - -
    • - For MacOS, to use wget: first, install brew; then, install wget with - brew install wget. - -

      - -

      -
      • - -
    - -
    - -
    - - Elasticsearch 2.1.1 - - -

    - Elasticsearch 2.1.1 requires a special, compatible version of the plug-in. This can be obtained - here. -

    - -

    - Use of this version of the plug-in requires that Java security-policy permissions be - appropriately set. See Advanced Settings. -

    - -
    - -
    - - Versioning and Compatibility - - -

    - The following table shows the supported version-combinations for the three principal components: -

    - - - Version Compatibility - - - - - - - - Plug-in - Couchbase - Elasticsearch - - - - - - master - 2.5.x to 4.x - 2.4.0 - - - - 2.4.0.0-alpha2 - 2.5.x to 4.x - 5.0.0.-alpha2 - - - - 2.2.x.y - 2.5.x to 4.x - 2.x.y - - - - 2.1 - 2.5.x to 4.x - 1.3.0 to 1.7.x - - - - 2.0 - 3.x, 2.5.x - 1.3.0 - - - - -
    - -
    - - - -     diff --git a/content/connectors/elasticsearch-2.2/install-and-config.html b/content/connectors/elasticsearch-2.2/install-and-config.html new file mode 100644 index 0000000000..44aea85a14 --- /dev/null +++ b/content/connectors/elasticsearch-2.2/install-and-config.html @@ -0,0 +1,15 @@ + + + +Redirecting… + + + + +
    +

    Redirecting…

    + Click here if you are not redirected. + +
    + + \ No newline at end of file diff --git a/content/connectors/elasticsearch-2.2/overview.html b/content/connectors/elasticsearch-2.2/overview.html new file mode 100644 index 0000000000..44042a6855 --- /dev/null +++ b/content/connectors/elasticsearch-2.2/overview.html @@ -0,0 +1,15 @@ + + + + Redirecting… + + + + +
    +

    Redirecting…

    + Click here if you are not redirected. + +
    + + diff --git a/content/connectors/elasticsearch-2.2/performance-tuning.html b/content/connectors/elasticsearch-2.2/performance-tuning.html new file mode 100644 index 0000000000..c24e5ca22e --- /dev/null +++ b/content/connectors/elasticsearch-2.2/performance-tuning.html @@ -0,0 +1,15 @@ + + + + Redirecting… + + + + +
    +

    Redirecting…

    + Click here if you are not redirected. + +
    + + diff --git a/content/connectors/elasticsearch-2.2/release-notes.dita b/content/connectors/elasticsearch-2.2/release-notes.dita deleted file mode 100644 index c8035106c8..0000000000 --- a/content/connectors/elasticsearch-2.2/release-notes.dita +++ /dev/null @@ -1,157 +0,0 @@ - - - - Release Notes - - - This section provides a set of Release Notes for successive versions of the - Elasticsearch Transport Plug-in, provided by Couchbase. Each set of notes - provide details of changes and additions that have been made. - - - - -
    - - Elasticsearch Plug-in 2.2 - - -

    - This release note applies to the 2.2 version of the Elasticsearch Transport Plug-in (February 2017). - It adds a number of bug fixes. See Installation - and Configuration for versioning and compatibility information. -

    -
    - -
    - Elasticsearch Plug-in 2.1.1 - -

    This release note applies to the 2.1.1 version of the Elasticsearch Transport Plug-in (September 2015). - It adds compatibility with newer Elasticsearch versions up to 1.7.x, - multiple new features, and quite a few bug fixes, including several that solve - issues found in 2.1.0. In particular, this release fixes a long-standing bug with an - incorrect concurrent bulk request counter, which could eventually cause the plug-in - to stop accepting requests from Couchbase Server altogether.

    -

    Some configuration option-names changed.

    -

    - - Version Compatibility - - - - - - - Elasticsearch plug-in version - Couchbase versions - Elasticsearch versions - - - - - 2.1.1 - 2.5.x - 4.x - 1.3.0 - 1.7.x - - - -
    -

    - -
    - -
    - Elasticsearch Plug-in 2.0 -

    This release note is for the Elasticsearch plug-in release 2.0 GA (October 2014). - Elasticsearch plug-in version 2.0 is compatible with:

      -
    • Elasticsearch 1.3.0.
      • -
    • Couchbase Server 3.0
      • -
    • Couchbase Server 2.5.x (backward compatible)
      • -

    - -

    The new feature(s) available in Elasticsearch Plugin v2.0:

      -
    • Support more than one document type in Elasticsearch. (MB-12284)
      • -

    - -

    The following are known issues:

    -
      -
    • The att_reason value for non-JSON documents changed from non-JSON - mode to invalid_json. If a Couchbase cluster has a lot of deletes, the Elasticsearch - log could fill up with a lot of messages. (CBES-31)
      • -
    - -
    - - -
    - Elasticsearch Plug-in 1.3.0 -

    This release note is for the Elasticsearch plug-in release 1.3.0 GA (April 2014). This - release is compatible only with Elasticsearch 1.0.1.

    -

    This release is compatible with Couchbase Server 2.5.x, and it is backward compatible - with earlier 2.x releases.

    -
      -
    • Support for new XDCR checkpoint protocol. (CBES-26)
      • -
    • Fixed failure handling due to bounded queue with Elasticsearch 1.x. (CBES-27)
      • -
    -
    - - -
    - Elasticsearch Plug-in 1.2.0 -

    This release note is for the Elasticsearch plug-in release 1.2.0 GA (October 2013). This - release adds compatibility with Elasticsearch 0.90.5.

    -

    This release is compatible with Couchbase Server 2.2, and it is backward compatible with - earlier 2.x releases.

    -
    - - -
    - Elasticsearch Plug-in 1.1.0 -

    This release note is for the Elasticsearch plug-in release 1.1.0 GA (August 2013). This - release adds compatibility with Elasticsearch 0.90.2.

    -
    - - -
    - Elasticsearch Plug-in 1.0.0 -

    This release note is for the Elasticsearch plug-in release 1.0.0 GA (February 2013). - This is the first general availability (GA) release. It contains the following - enhancements and bug fixes:

    -
      -
    • Now compatible with version 0.20.2 of Elasticsearch.
      • -
    • Now supports document expiration using Elasticsearch TTL.
      • -
    • Now supports XDCR conflict resolution to reduce bandwidth usage in some cases.
      • -
    • Fixed Couchbase index template to allow searching on the document metadata.
      • -
    • Fixed data corruption under high load. (CBES-11)
      • -
    • Fixed recognition of non-JSON documents. (CBES-11)
      • -
    • Improved log information when indexing stub documents.
      • -
    -
    - - -
    - Elasticsearch Plug-in 1.0.0 Beta -

    This is the beta release of the Couchbase plug-in for Elasticsearch 1.0.0 Beta (February - 2013).

    - - - -
    - - - - -     diff --git a/content/connectors/elasticsearch-2.2/release-notes.html b/content/connectors/elasticsearch-2.2/release-notes.html new file mode 100644 index 0000000000..9023bf0eb0 --- /dev/null +++ b/content/connectors/elasticsearch-2.2/release-notes.html @@ -0,0 +1,15 @@ + + + + Redirecting… + + + + +
    +

    Redirecting…

    + Click here if you are not redirected. + +
    + + diff --git a/content/connectors/elasticsearch-2.2/troubleshooting.html b/content/connectors/elasticsearch-2.2/troubleshooting.html new file mode 100644 index 0000000000..376f25d62c --- /dev/null +++ b/content/connectors/elasticsearch-2.2/troubleshooting.html @@ -0,0 +1,15 @@ + + + + Redirecting… + + + + +
    +

    Redirecting…

    + Click here if you are not redirected. + +
    + + diff --git a/content/connectors/elasticsearch-2.2/working-with-documents.html b/content/connectors/elasticsearch-2.2/working-with-documents.html new file mode 100644 index 0000000000..644cac32f6 --- /dev/null +++ b/content/connectors/elasticsearch-2.2/working-with-documents.html @@ -0,0 +1,15 @@ + + + + Redirecting… + + + + +
    +

    Redirecting…

    + Click here if you are not redirected. + +
    + + diff --git a/content/connectors/elasticsearch-2.2/advanced-settings.dita b/content/connectors/elasticsearch/advanced-settings.dita similarity index 100% rename from content/connectors/elasticsearch-2.2/advanced-settings.dita rename to content/connectors/elasticsearch/advanced-settings.dita diff --git a/content/connectors/elasticsearch/elastic.ditamap b/content/connectors/elasticsearch/elastic.ditamap new file mode 100644 index 0000000000..4e53a67a8a --- /dev/null +++ b/content/connectors/elasticsearch/elastic.ditamap @@ -0,0 +1,13 @@ + + + + Elasticsearch Plug-in + + + + + + + + + diff --git a/content/connectors/elasticsearch/getting-started.html b/content/connectors/elasticsearch/getting-started.html new file mode 100644 index 0000000000..e4c2ec81f4 --- /dev/null +++ b/content/connectors/elasticsearch/getting-started.html @@ -0,0 +1,507 @@ + + + + + Getting Started + + + +
    + + + + + + + +
    + + + + +
    + +
    + Platform: + +
    +
    + + + +

    A Couchbase-Elasticsearch data-replication system consists of three principal components:

    + +
      +
    • a Couchbase Server cluster of one or more nodes
      • +
    • an Elasticsearch cluster of one or more nodes
      • +
    • the Elasticsearch Transport Plug-in, installed in the Elasticsearch environment.
      • +
    + +

    This section provides step-by-step instructions to install the Couchbase Elasticsearch plugin.

    +

    Plugin pre-requisites

    + + +

    Verify the Elasticsearch cluster is up and running (the default port is 9200).

    + 
    $ curl localhost:9200
+
+{
+  "name" : "K3RqW4F",
+  "cluster_name" : "elasticsearch",
+  "cluster_uuid" : "Bw-Ta0wDTcekzQIhXZHGkg",
+  "version" : {
+    "number" : "5.6.5",
+    "build_hash" : "6a37571",
+    "build_date" : "2017-12-04T07:50:10.466Z",
+    "build_snapshot" : false,
+    "lucene_version" : "6.6.1"
+  },
+  "tagline" : "You Know, for Search"
+}
+
    +

    Verify that Couchbase Server is running.

    + 
    $ curl localhost:8092
+
+{"couchdb":"Welcome","version":"v4.5.1-60-g3cf258d","couchbase":"5.0.2-5506-community"}
+

    Installation

    +

    Navigate to the Elasticsearch installation directory.

    + 
    $ cd /usr/share/elasticsearch
+
    +

    Download and install the plug-in package.

    + 
    $ bin/elasticsearch-plugin install https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/3.0.0-cypress/elasticsearch-transport-couchbase-3.0.0-cypress-es5.6.4.zip
+
    +

    Replace the plugin URL with the one that matches your Elasticsearch version; all URLs can be found on the releases page.

    + +
    +

    Note: If you're using Elasticsearch 2.x, the command to install plugins is bin/plugin instead of bin/elastichsearch-plugin.

    +
    + +

    When the installation is successful, the message "Installed transport-couchbase into /usr/share/elasticsearch/plugins/transport/couchbase" will be logged.

    +

    Configuration

    +

    Open the Elasticsearch configuration file (/etc/elasticsearch/elasticsearch.yml) and add the following to the end of the file.

    + 
    couchbase.username: <USERNAME>
+couchbase.password: <PASSWORD>
+couchbase.maxConcurrentRequests: 1024
+
    +

    The username and password credentials will be used again later when configuring Couchbase Server.

    + +

    Still in the /usr/share/elasticsearch directory, configure the Elasticsearch plugin with the following curl command.

    + 
    $ curl -X PUT http://localhost:9200/_template/couchbase -d @plugins/transport-couchbase/couchbase_template.json
+
    +

    When successful, the configuration-routine provides the following response: {"acknowledged":true}.

    + +

    Create an Elasticsearch index to receive the data from the Couchbase bucket. Later, when configuring XDCR, the "remote bucket name" should match the Elasticsearch index name.

    + 
    $ curl -X PUT http://localhost:9200/travel-sample
+
+{"acknowledged":true}
+
    +

    Now to configure Couchbase Server, open the Couchbase Web Console and select XDCR > Add Remote Cluster.

    + +

    + +

    In the dialog, enter the Cluster Name of your choice, the IP/hostname and port number where the Elasticsearch cluster is running (the Elasticsearch plugin listens on port 9091 by default) and the Username/Password previously stored in elasticsearch.yml.

    + +

    + +

    Next, select the Add Replication option.

    + +

    + +

    On the pop-up window, enter the origin bucket, the remote cluster, and remote bucket (in this case it's the index that was created earlier). Also make sure that the XDCR Protocol version is set to 1 and that the XDCR Optimistic Replication Threshold is set to the maximum value of 20971520 for optimal performance. Then click Save.

    + +

    + +

    Now that the replication is up and running, you can run full-text search queries on the Elasticsearch node.

    + 
    $ curl localhost:9200/travel-sample/_search?q=san+francisco
+
+{
+    "took":5,
+    "timed_out":false,
+    "_shards":{"total":5,"successful":5,"skipped":0,"failed":0},
+    "hits":{
+        "total":1599,
+        "max_score":11.965878,
+        "hits":[
+            {"_index":"travel-sample","_type":"couchbaseDocument","_id":"landmark_36047","_score":11.965878,"_source":{"meta":{"rev":"1-1508c18bdbb400000000000002000000","flags":33554432,"expiration":0,"id":"landmark_36047"}}},
+            {"_index":"travel-sample","_type":"couchbaseDocument","_id":"landmark_25611","_score":11.905596,"_source":{"meta":{"rev":"1-1508c18bb43400000000000002000000","flags":33554432,"expiration":0,"id":"landmark_25611"}}},
+            {"_index":"travel-sample","_type":"couchbaseDocument","_id":"landmark_25712","_score":11.905596,"_source":{"meta":{"rev":"1-1508c18bb61e00000000000002000000","flags":33554432,"expiration":0,"id":"landmark_25712"}}}
+            ...
+        ]
+    }
+}
+
    +

    Let's take a moment to discuss some properties in the response object:

    + +
      +
    • The took parameter indicates the number of milliseconds required for the search; while fields within the _shards object indicate how many Elasticsearch shards were available for search, how many were accessed successfully, and how many unsuccessfully.
      • +
    • The total field indicates the total number of items. A max_score is provided, to indicate Elasticsearch’s estimate of the relevance of each search-hit. Note that the source object contains only metadata, rather than a document’s entire contents: this is because the contents, if and when required, can more rapidly be retrieved from Couchbase itself; using the document ID that is the value of the _id field.
      • +
    +

    Deployment Considerations

    +

    If you are working with an ElasticSearch cluster, it is recommended that you install the ElasticSearch Transport Plugin on every node. At minimum, you can install it on a single node. The plugin checks which nodes it's installed on and reports the public host addresses of those back to Couchbase. XDCR is then directed to every node where the plugin is installed.

    + +

    Optionally, if your ElasticSearch cluster has separate data and client nodes, you can elect to install the ElasticSearch Transport Plugin on just one set of nodes. By installing on the data nodes only, you can reduce the amount of routing that needs to be done. Alternatively, installing the plugin on only the client nodes will offload CPU from the data nodes.

    + + + + + + + + + + + +
    + + + +
    + + diff --git a/content/connectors/elasticsearch/images/addreplication-popup.png b/content/connectors/elasticsearch/images/addreplication-popup.png new file mode 100644 index 0000000000..a9159bc68d Binary files /dev/null and b/content/connectors/elasticsearch/images/addreplication-popup.png differ diff --git a/content/connectors/elasticsearch/images/addreplication.png b/content/connectors/elasticsearch/images/addreplication.png new file mode 100644 index 0000000000..f8a561581e Binary files /dev/null and b/content/connectors/elasticsearch/images/addreplication.png differ diff --git a/content/connectors/elasticsearch-2.2/images/advancedSettings.png b/content/connectors/elasticsearch/images/advancedSettings.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/advancedSettings.png rename to content/connectors/elasticsearch/images/advancedSettings.png diff --git a/content/connectors/elasticsearch-2.2/images/beerSampleDocuments.png b/content/connectors/elasticsearch/images/beerSampleDocuments.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/beerSampleDocuments.png rename to content/connectors/elasticsearch/images/beerSampleDocuments.png diff --git a/content/connectors/elasticsearch-2.2/images/beerSampleDocumentsLookUpField.png b/content/connectors/elasticsearch/images/beerSampleDocumentsLookUpField.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/beerSampleDocumentsLookUpField.png rename to content/connectors/elasticsearch/images/beerSampleDocumentsLookUpField.png diff --git a/content/connectors/elasticsearch-2.2/images/beerSampleDocumentsLookUpResults.png b/content/connectors/elasticsearch/images/beerSampleDocumentsLookUpResults.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/beerSampleDocumentsLookUpResults.png rename to content/connectors/elasticsearch/images/beerSampleDocumentsLookUpResults.png diff --git a/content/connectors/elasticsearch-2.2/images/beerSampleLine.png b/content/connectors/elasticsearch/images/beerSampleLine.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/beerSampleLine.png rename to content/connectors/elasticsearch/images/beerSampleLine.png diff --git a/content/connectors/elasticsearch-2.2/images/blank.png b/content/connectors/elasticsearch/images/blank.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/blank.png rename to content/connectors/elasticsearch/images/blank.png diff --git a/content/connectors/elasticsearch-2.2/images/clusterReferenceCreated.png b/content/connectors/elasticsearch/images/clusterReferenceCreated.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/clusterReferenceCreated.png rename to content/connectors/elasticsearch/images/clusterReferenceCreated.png diff --git a/content/connectors/elasticsearch-2.2/images/codeExampleDiagram.png b/content/connectors/elasticsearch/images/codeExampleDiagram.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/codeExampleDiagram.png rename to content/connectors/elasticsearch/images/codeExampleDiagram.png diff --git a/content/connectors/elasticsearch-2.2/images/couchbaseConsoleInitial.png b/content/connectors/elasticsearch/images/couchbaseConsoleInitial.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/couchbaseConsoleInitial.png rename to content/connectors/elasticsearch/images/couchbaseConsoleInitial.png diff --git a/content/connectors/elasticsearch-2.2/images/createClusterReferenceDialog.png b/content/connectors/elasticsearch/images/createClusterReferenceDialog.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/createClusterReferenceDialog.png rename to content/connectors/elasticsearch/images/createClusterReferenceDialog.png diff --git a/content/connectors/elasticsearch-2.2/images/createReplicationDialog.png b/content/connectors/elasticsearch/images/createReplicationDialog.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/createReplicationDialog.png rename to content/connectors/elasticsearch/images/createReplicationDialog.png diff --git a/content/connectors/elasticsearch-2.2/images/create_cluster_ref.png b/content/connectors/elasticsearch/images/create_cluster_ref.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/create_cluster_ref.png rename to content/connectors/elasticsearch/images/create_cluster_ref.png diff --git a/content/connectors/elasticsearch-2.2/images/dataBucketsTab.png b/content/connectors/elasticsearch/images/dataBucketsTab.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/dataBucketsTab.png rename to content/connectors/elasticsearch/images/dataBucketsTab.png diff --git a/content/connectors/elasticsearch-2.2/images/demoUIfullQueryResults.png b/content/connectors/elasticsearch/images/demoUIfullQueryResults.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/demoUIfullQueryResults.png rename to content/connectors/elasticsearch/images/demoUIfullQueryResults.png diff --git a/content/connectors/elasticsearch-2.2/images/demoUIinitial.png b/content/connectors/elasticsearch/images/demoUIinitial.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/demoUIinitial.png rename to content/connectors/elasticsearch/images/demoUIinitial.png diff --git a/content/connectors/elasticsearch-2.2/images/demoUIqueryESbutton.png b/content/connectors/elasticsearch/images/demoUIqueryESbutton.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/demoUIqueryESbutton.png rename to content/connectors/elasticsearch/images/demoUIqueryESbutton.png diff --git a/content/connectors/elasticsearch-2.2/images/demoUIradioButtons.png b/content/connectors/elasticsearch/images/demoUIradioButtons.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/demoUIradioButtons.png rename to content/connectors/elasticsearch/images/demoUIradioButtons.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_components.png b/content/connectors/elasticsearch/images/elastic_components.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_components.png rename to content/connectors/elasticsearch/images/elastic_components.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_get_doc.png b/content/connectors/elasticsearch/images/elastic_get_doc.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_get_doc.png rename to content/connectors/elasticsearch/images/elastic_get_doc.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_head.png b/content/connectors/elasticsearch/images/elastic_head.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_head.png rename to content/connectors/elasticsearch/images/elastic_head.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_model_int.png b/content/connectors/elasticsearch/images/elastic_model_int.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_model_int.png rename to content/connectors/elasticsearch/images/elastic_model_int.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_model_str.png b/content/connectors/elasticsearch/images/elastic_model_str.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_model_str.png rename to content/connectors/elasticsearch/images/elastic_model_str.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_montor_xdcr.png b/content/connectors/elasticsearch/images/elastic_montor_xdcr.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_montor_xdcr.png rename to content/connectors/elasticsearch/images/elastic_montor_xdcr.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_replicate.png b/content/connectors/elasticsearch/images/elastic_replicate.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_replicate.png rename to content/connectors/elasticsearch/images/elastic_replicate.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_replicate_2.png b/content/connectors/elasticsearch/images/elastic_replicate_2.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_replicate_2.png rename to content/connectors/elasticsearch/images/elastic_replicate_2.png diff --git a/content/connectors/elasticsearch-2.2/images/elastic_xdcr_ref_fail.png b/content/connectors/elasticsearch/images/elastic_xdcr_ref_fail.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elastic_xdcr_ref_fail.png rename to content/connectors/elasticsearch/images/elastic_xdcr_ref_fail.png diff --git a/content/connectors/elasticsearch-2.2/images/elasticsearchConsole.png b/content/connectors/elasticsearch/images/elasticsearchConsole.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/elasticsearchConsole.png rename to content/connectors/elasticsearch/images/elasticsearchConsole.png diff --git a/content/connectors/elasticsearch-2.2/images/esCbArch_components.png b/content/connectors/elasticsearch/images/esCbArch_components.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/esCbArch_components.png rename to content/connectors/elasticsearch/images/esCbArch_components.png diff --git a/content/connectors/elasticsearch-2.2/images/esCouchbaseArch.png b/content/connectors/elasticsearch/images/esCouchbaseArch.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/esCouchbaseArch.png rename to content/connectors/elasticsearch/images/esCouchbaseArch.png diff --git a/content/connectors/elasticsearch-2.2/images/itemCount.png b/content/connectors/elasticsearch/images/itemCount.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/itemCount.png rename to content/connectors/elasticsearch/images/itemCount.png diff --git a/content/connectors/elasticsearch-2.2/images/outboundXDCRops.png b/content/connectors/elasticsearch/images/outboundXDCRops.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/outboundXDCRops.png rename to content/connectors/elasticsearch/images/outboundXDCRops.png diff --git a/content/connectors/elasticsearch-2.2/images/pickRemoteCluster.png b/content/connectors/elasticsearch/images/pickRemoteCluster.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/pickRemoteCluster.png rename to content/connectors/elasticsearch/images/pickRemoteCluster.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolAuthPermission.png b/content/connectors/elasticsearch/images/policyToolAuthPermission.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolAuthPermission.png rename to content/connectors/elasticsearch/images/policyToolAuthPermission.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolClickOK.png b/content/connectors/elasticsearch/images/policyToolClickOK.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolClickOK.png rename to content/connectors/elasticsearch/images/policyToolClickOK.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolFirstEntry.png b/content/connectors/elasticsearch/images/policyToolFirstEntry.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolFirstEntry.png rename to content/connectors/elasticsearch/images/policyToolFirstEntry.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolInitial.png b/content/connectors/elasticsearch/images/policyToolInitial.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolInitial.png rename to content/connectors/elasticsearch/images/policyToolInitial.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolModifyPrincipals.png b/content/connectors/elasticsearch/images/policyToolModifyPrincipals.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolModifyPrincipals.png rename to content/connectors/elasticsearch/images/policyToolModifyPrincipals.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolPermissionsDialog.png b/content/connectors/elasticsearch/images/policyToolPermissionsDialog.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolPermissionsDialog.png rename to content/connectors/elasticsearch/images/policyToolPermissionsDialog.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolPolicyEnter01.png b/content/connectors/elasticsearch/images/policyToolPolicyEnter01.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolPolicyEnter01.png rename to content/connectors/elasticsearch/images/policyToolPolicyEnter01.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolSocketPermission.png b/content/connectors/elasticsearch/images/policyToolSocketPermission.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolSocketPermission.png rename to content/connectors/elasticsearch/images/policyToolSocketPermission.png diff --git a/content/connectors/elasticsearch-2.2/images/policyTooldone.png b/content/connectors/elasticsearch/images/policyTooldone.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyTooldone.png rename to content/connectors/elasticsearch/images/policyTooldone.png diff --git a/content/connectors/elasticsearch-2.2/images/policyToolpolicyEntryFirstEntry.png b/content/connectors/elasticsearch/images/policyToolpolicyEntryFirstEntry.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/policyToolpolicyEntryFirstEntry.png rename to content/connectors/elasticsearch/images/policyToolpolicyEntryFirstEntry.png diff --git a/content/connectors/elasticsearch/images/remotecluster.png b/content/connectors/elasticsearch/images/remotecluster.png new file mode 100644 index 0000000000..a6a4113f14 Binary files /dev/null and b/content/connectors/elasticsearch/images/remotecluster.png differ diff --git a/content/connectors/elasticsearch-2.2/images/replicationInProgress.png b/content/connectors/elasticsearch/images/replicationInProgress.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/replicationInProgress.png rename to content/connectors/elasticsearch/images/replicationInProgress.png diff --git a/content/connectors/elasticsearch-2.2/images/selectCreateClusterReference.png b/content/connectors/elasticsearch/images/selectCreateClusterReference.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/selectCreateClusterReference.png rename to content/connectors/elasticsearch/images/selectCreateClusterReference.png diff --git a/content/connectors/elasticsearch-2.2/images/selectVersion1.png b/content/connectors/elasticsearch/images/selectVersion1.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/selectVersion1.png rename to content/connectors/elasticsearch/images/selectVersion1.png diff --git a/content/connectors/elasticsearch-2.2/images/selectXDCR.png b/content/connectors/elasticsearch/images/selectXDCR.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/selectXDCR.png rename to content/connectors/elasticsearch/images/selectXDCR.png diff --git a/content/connectors/elasticsearch-2.2/images/view_replication.png b/content/connectors/elasticsearch/images/view_replication.png similarity index 100% rename from content/connectors/elasticsearch-2.2/images/view_replication.png rename to content/connectors/elasticsearch/images/view_replication.png diff --git a/content/connectors/elasticsearch/images/xdcrInitial.png b/content/connectors/elasticsearch/images/xdcrInitial.png new file mode 100644 index 0000000000..d7e733ebd7 Binary files /dev/null and b/content/connectors/elasticsearch/images/xdcrInitial.png differ diff --git a/content/connectors/elasticsearch-2.2/overview.dita b/content/connectors/elasticsearch/overview.dita similarity index 99% rename from content/connectors/elasticsearch-2.2/overview.dita rename to content/connectors/elasticsearch/overview.dita index d7dc83af42..00576bd76c 100644 --- a/content/connectors/elasticsearch-2.2/overview.dita +++ b/content/connectors/elasticsearch/overview.dita @@ -2,8 +2,8 @@ -Elasticsearch Plug-in 2.2 - +Elasticsearch Plug-in + This document explains how to use the Elasticsearch Transport Plugin to coordinate data-searches across Elasticsearch and Couchbase Server. It explains the system architecture, provides diff --git a/content/connectors/elasticsearch-2.2/performance-tuning.dita b/content/connectors/elasticsearch/performance-tuning.dita similarity index 100% rename from content/connectors/elasticsearch-2.2/performance-tuning.dita rename to content/connectors/elasticsearch/performance-tuning.dita diff --git a/content/connectors/elasticsearch/release-notes.html b/content/connectors/elasticsearch/release-notes.html new file mode 100644 index 0000000000..aad0ae45e2 --- /dev/null +++ b/content/connectors/elasticsearch/release-notes.html @@ -0,0 +1,783 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Elasticsearch Plugin - Release Notes + + + + + + + + + + + + + + + +
    +
    +
    + +
    + + +
    +
    +
    + +
    +
    +
    + + + + + + + + + + Edit on GitHub + + +
    + + + + +
    + +
    + Platform: + +
    +
    + + + +

    This section provides a set of Release Notes for successive versions of the Elasticsearch Transport Plug-in, provided by Couchbase. Each set of notes provide details of changes and additions that have been made.

    +

    Version Compatibility

    +

    The GitHub releases page lists all the plugin variants. In a given release, the plugin's filename is of the form elasticsearch-transport-couchbase-${PLUGIN_VERSION}-alder-es${ES_VERSION}.zip. It is imperative to use the file where ES_VERSION matches your Elasticsearch cluster version. For example, to use the 3.0.0 version of the plugin with Elasticsearch 5.6.4, you will need to download elasticsearch-transport-couchbase-3.0.0-cypress-es5.6.4.zip.

    +

    Elasticsearch Plug-in 3.0

    +

    With this release, the Couchbase plugin for Elasticsearch moves to a new versioning and branch management strategy that allows for simultaneous releases across ES versions (see Version Compatibility). All the versions are expected to work and are supported. However, versions which have received more testing and are officially supported are:

    + +
      +
    • 3.0.0-cypress-es5.6.4
      • +
    • 3.0.0-birch-es5.2.2
      • +
    • 3.0.0-alder-es2.4.0
      • +
    + +

    Don't be alarmed by the major version bump; upgrading from version 2.x of the plugin should be seamless, and is recommended for all users. The code has just been relabeled for easier maintenance.

    +

    New Feature

    +
      +
    • Adds the couchbase.pipeline config for specifying the ingestion pipeline.
      • +
    +

    Enhancements

    +
      +
    • Improves logging for indexing errors.
      • +
    • Uses Dropwizard Metrics to collect and log richer stats.
      • +
    +

    Bug Fixes

    +
      +
    • CBES-48: Connection counter leak could cause spurious TooManyConcurrentConnections errors.
      • +
    • #153: ClassCastException error when the TTL is of type Long instead of Integer.
      • +
    +

    Known issues

    +

    The Elasticsearch Plug-in does not support IPv6. So to use the plugin, the Couchbase Server and Elasticsearch clusters will need to run on instances which are addressable with IPv4.

    +

    Elasticsearch Plug-in 2.2

    +

    This release note applies to the 2.2 version of the Elasticsearch Transport Plug-in (February 2017). It adds a number of bug fixes. See Installation and Configuration for versioning and compatibility information.

    +

    Elasticsearch Plug-in 2.1.1

    +

    This release note applies to the 2.1.1 version of the Elasticsearch Transport Plug-in (September 2015). It adds compatibility with newer Elasticsearch versions up to 1.7.x, multiple new features, and quite a few bug fixes, including several that solve issues found in 2.1.0. In particular, this release fixes a long-standing bug with an incorrect concurrent bulk request counter, which could eventually cause the plug-in to stop accepting requests from Couchbase Server altogether.

    + +

    Some configuration option-names changed.

    + + + + + + + + + + + + + +
    Elasticsearch plug-in versionCouchbase versionsElasticsearch versions
    2.1.12.5.x - 4.x1.3.0 - 1.7.x
    +

    Elasticsearch Plug-in 2.0

    +

    This release note is for the Elasticsearch plug-in release 2.0 GA (October 2014). Elasticsearch plug-in version 2.0 is compatible with:

    + +
      +
    • Elasticsearch 1.3.0.
      • +
    • Couchbase Server 3.0
      • +
    • Couchbase Server 2.5.x (backward compatible)
      • +
    + +

    The new feature(s) available in Elasticsearch Plugin v2.0:

    + +
      +
    • Support more than one document type in Elasticsearch. (MB-12284)
      • +
    + +

    The following are known issues:

    + +
      +
    • The att_reason value for non-JSON documents changed from non-JSON mode to invalid_json. If a Couchbase cluster has a lot of deletes, the Elasticsearch log could fill up with a lot of messages. (CBES-31)
      • +
    +

    Elasticsearch Plug-in 1.3.0

    +

    This release note is for the Elasticsearch plug-in release 1.3.0 GA (April 2014). This release is compatible only with Elasticsearch 1.0.1.

    + +

    This release is compatible with Couchbase Server 2.5.x, and it is backward compatible with earlier 2.x releases.

    + +
      +
    • Support for new XDCR checkpoint protocol. (CBES-26)
      • +
    • Fixed failure handling due to bounded queue with Elasticsearch 1.x. (CBES-27)
      • +
    +

    Elasticsearch Plug-in 1.2.0

    +

    This release note is for the Elasticsearch plug-in release 1.2.0 GA (October 2013). This release adds compatibility with Elasticsearch 0.90.5.

    + +

    This release is compatible with Couchbase Server 2.2, and it is backward compatible with earlier 2.x releases.

    +

    Elasticsearch Plug-in 1.1.0

    +

    This release note is for the Elasticsearch plug-in release 1.1.0 GA (August 2013). This release adds compatibility with Elasticsearch 0.90.2.

    +

    Elasticsearch Plug-in 1.0.0

    +

    This release note is for the Elasticsearch plug-in release 1.0.0 GA (February 2013). This is the first general availability (GA) release. It contains the following enhancements and bug fixes:

    + +
      +
    • Now compatible with version 0.20.2 of Elasticsearch.
      • +
    • Now supports document expiration using Elasticsearch TTL.
      • +
    • Now supports XDCR conflict resolution to reduce bandwidth usage in some cases.
      • +
    • Fixed Couchbase index template to allow searching on the document metadata.
      • +
    • Fixed data corruption under high load. (CBES-11)
      • +
    • Fixed recognition of non-JSON documents. (CBES-11)
      • +
    • Improved log information when indexing stub documents.
      • +
    +

    Elasticsearch Plug-in 1.0.0 Beta

    +

    This is the beta release of the Couchbase plug-in for Elasticsearch 1.0.0 Beta (February 2013).

    + + + + + + + + + + + +
    + + + +
    +
    +
    +
    +
    + +
    + +
    +
    +
    + + + + + + + + + + + + \ No newline at end of file diff --git a/content/connectors/elasticsearch-2.2/troubleshooting.dita b/content/connectors/elasticsearch/troubleshooting.dita similarity index 98% rename from content/connectors/elasticsearch-2.2/troubleshooting.dita rename to content/connectors/elasticsearch/troubleshooting.dita index 8425a3f8b7..24fbc4f328 100644 --- a/content/connectors/elasticsearch-2.2/troubleshooting.dita +++ b/content/connectors/elasticsearch/troubleshooting.dita @@ -110,7 +110,7 @@ Note that a plug-in version-incompatibility may produce an error message such as the following: diff --git a/content/connectors/elasticsearch-2.2/working-with-documents.dita b/content/connectors/elasticsearch/working-with-documents.dita similarity index 100% rename from content/connectors/elasticsearch-2.2/working-with-documents.dita rename to content/connectors/elasticsearch/working-with-documents.dita diff --git a/content/connectors/hadoop-1.2/hadoop.dita b/content/connectors/hadoop-1.2/hadoop.dita index 8d96d21ac6..01a0d0b6ef 100644 --- a/content/connectors/hadoop-1.2/hadoop.dita +++ b/content/connectors/hadoop-1.2/hadoop.dita @@ -1,263 +1,9 @@ - Hadoop Connector 1.2 - The Couchbase Hadoop Connector allows you to connect to Couchbase Server 2.5, 3.x, or 4.x; - in order to stream keys into either Hadoop Distributed File System (HDFS) or Hive, for processing with - Hadoop. + Hadoop Connector + -

    If you have previously used Apache - Sqoop with other databases, using the Couchbase Hadoop Connector should be straightforward: - because it uses a similar command-line argument-structure (although some of the arguments themselves might seem - slightly different, because Couchbase has a very different structure than does a typical - RDBMS).

    - -
    - Getting Started -

    Download the Couchbase Hadoop Connector version 1.2.0, from http://packages.couchbase.com/clients/connectors/couchbase-hadoop-plugin-1.2.0.zip.

    -

    The Couchbase Hadoop Connector is supported on Cloudera 5. Cloudera has certified the Couchbase Hadoop - Connector 1.2 release for Cloudera 5.

    -

    The Couchbase Hadoop Connector is supported on Hortonworks Data Platform (HDP) 2.2. - Hortonworks has certified the Couchbase Hadoop Connector 1.2 release for HDP - 2.2.

    -
    - -
    - Installation

    You can install the Couchbase Hadoop Connector either by - running a script, or by manually copying the files to specified directories within - your Sqoop installation. The distribution-package contains a set of files that needs - to be copied into your Sqoop installation; and a script that copies the files for you, - if you provide the path to the Sqoop installation.

    The following table - describes the files in the Couchbase Hadoop Connector distribution, and lists where - each file is installed. In the installation location column, - $sqoop_home represents the path to your Sqoop installation.

    - - Files in the Couchbase Hadoop Connector - package - - - - - - - File name - Description - Installation location - - - - - couchbase-client-1.4.4.bundled.jar - A library-dependency of the connector that handles the basic - communications with the Couchbase cluster - $sqoop_home/lib - - - couchbase-config.xml - A property file used to register a - ManagerFactory for the connector with - Sqoop - $sqoop_home/conf - - - couchbase-hadoop-plugin-1.2.0.jar - The Couchbase Hadoop Connector - $sqoop_home/lib - - - couchbase-manager.xml - A property file that tells Sqoop where the - ManagerFactory defined in the - couchsqoop-config.xml resides - $sqoop_home/conf/managers.d - - - install.sh - The Couchbase Hadoop Connector installation script - Not applicable - - - jettison-1.1.jar - A dependency of the Couchbase client - $sqoop_home/lib - - - netty-3.5.5.Final.jar - A dependency of the Couchbase client - $sqoop_home/lib - - - spymemcached-2.11.4.jar - A library-dependency of the Couchbase client that provides - networking and core protocol handling for data transfer - $sqoop_home/lib - - - -
    -

    -

    - - Script-based Installation -

    Script-based installation is done through the use of the - install.sh script, which comes with the connector-download. - The script takes one argument, which is the path to your Sqoop installation. The basic - command-format for invoking the script is: -

    shell> sh install.sh path_to_sqoop_home

    In an HDP - deployment, Sqoop is located at /usr/hdp/current/sqoop-client, - which you use as the path to the Sqoop installation. For HDP, invoke the - installation script as follows:

    shell> sh install.sh /usr/hdp/current/sqoop-client - Manual Installation -

    To install the Couchbase Hadoop Connector manually, copy each JAR and XML file listed - in the table Files in the - Couchbase Hadoop Connector package into the directory specified in the - installation-location column.

    - Uninstallation -

    Uninstallation of the connector requires removal of all of the files that were added - to Sqoop during installation. To uninstall the files, cd into - your Sqoop home directory, and execute the following - command:

    shell> rm lib/couchbase-hadoop-plugin-1.2.0.jar \ - lib/spymemcached-2.11.4.jar \ - lib/jettison-1.1.jar \ - lib/netty-3.5.5.Final.jar \ - lib/couchbase-client-1.4.4.bundled.jar \ - conf/couchbase-config.xml \ - conf/managers.d/couchbase-manager.xml -
    -
    - Using Sqoop

    The Couchbase Hadoop Connector can be used with a variety - of command-line tools provided by Sqoop. In this section, we discuss the usage of - each tool.

    - Tables -

    Since Sqoop is built for a relational model, it requires that the user specify a - table, to import and export into Couchbase. The Couchbase Hadoop Connector uses the - ‑‑table option, to specify the type of data stream - for importing and exporting into Couchbase.

    For exports, the user must enter a - value for the --table option, though what is entered will not be - used by the connector.

    For imports, the table command accepts two values; and - will exit reporting errors with invalid input.

      -
    • -

      DUMP—Causes all keys currently in Couchbase to be read into - HDFS. Any data-items received by the Couchbase cluster while this - command is running will also be passed along by the connector; meaning new or - changed items are part of the dump. However, items removed while the dump is - running will not be removed from the output.

      -
      • -
    • -

      BACKFILL_##—Streams all key-mutations, for a specified amount of - time (in minutes). For example, BACKFILL_5 means stream key mutations in the Couchbase server - for 5 minutes, and then stop the stream.

      -
      • -
    Connect string -

    A connect string option is required to connect to Couchbase. This can be specified - with --connect as an argument to the sqoop - command. The following example shows some connect - strings:

    http://10.2.1.55:8091/pools -http://10.2.1.55:8091/pools,http://10.2.1.56:8091/pools -

    When - creating your connect strings, replace the IP address above with the host name or IP - address of one or more nodes in your Couchbase Cluster. If you have multiple servers, - you can list them in a comma-separated list.

    - Connecting to different buckets -

    By default, the Couchbase Hadoop Connector connects to the default - bucket. If you want to connect to a bucket other than the default bucket, you can - specify the bucket-name by using the --username option. If the bucket - has a password, use the --password option, followed by the - password.

    Note that there are several variations on how the password may be - specified to Sqoop. The -P argument allows the password to be - supplied on the command line, and the --password-file argument - allows the password to be read from a file within HDFS.

    - Importing -

    Importing data to your cluster requires the use of the Sqoop import command followed - by the parameters --connect and --table.

    The - following example dumps all items from Couchbase into HDFS. Since the - Couchbase Java Client has support for a number of different data types, all values - are normalized to strings when being written to a Hadoop text - file.

    shell> sqoop import --connect http://10.2.1.55:8091/pools --table DUMP \ - --fields-terminated-by '\t' --escaped-by \\ --enclosed-by '\"'

    The - following example streams all item mutations from Couchbase into HDFS, for a - period of 10 minutes. -

    shell> sqoop import --connect http://10.2.1.55:8091/pools --table BACKFILL_10 \ - --fields-terminated-by '\t' --escaped-by \\ --enclosed-by '\"'

    In - both of the above examples, the delimiters for fields, new lines, and escape - have been explicitly specified. Note that Sqoop's default delimiters are comma - (,), for fields; and newline character - (\n), for records: these defaults should not be used when - field-data itself contains commas or newline characters, in case records become ambiguous or - unparsable.

    Sqoop - provides many more options to the import command than are covered in this document. - Run sqoop import help for a list of all options, and see the Sqoop - documentation for more details about these options.

    You have a number of - options for how to supply the password when accessing a bucket. The following - examples are equivalent for a bucket named mybucket, which uses the - password mypassword: -

    shell> sqoop import --username mybucket -P --verbose \ - --connect http://10.2.1.55:8091/pools --table DUMP \ - --fields-terminated-by '\t' --escaped-by \\ --enclosed-by '\"'shell> sqoop import --username mybucket --password mypassword --verbose \ - --connect http://10.2.1.55:8091/pools --table DUMP \ - --fields-terminated-by '\t' --escaped-by \\ --enclosed-by '\"'shell> sqoop import --username mybucket --password-file passwordfile \ - --verbose --connect http://10.2.1.55:8091/pools --table DUMP \ - --fields-terminated-by '\t' --escaped-by \\ --enclosed-by '\"'

    When the import job executes, it also generates - a .java source-code file; which can facilitate reading and writing - the records imported by other Hadoop MapReduce jobs. If, for instance, the job - was a DUMP, Sqoop generates a DUMP.java source-code - file.

    - Exporting -

    Exporting data to your cluster requires the use of the sqoop export - command, followed by the parameters --connect, - --export-dir, and --table.

    The following - example exports all records from the files in the HDFS directory specified by - --export-dir into - Couchbase.

    shell> sqoop export --connect http://10.2.1.55:8091/pools \ - --table couchbaseExportJob --export-dir data_for_export \ - --fields-terminated-by '\t' --escaped-by \\ --enclosed-by '\"'

    When the export job executes, it - also generates a .java source code file that shows how the data was - read. If, for instance, the job had the argument --table - couchbaseExportJob, Sqoop generates a - couchbaseExportJob.java source code file.

    - List table -

    Sqoop has a tool called list-tables. Couchbase does not have a - notion of tables, but we use DUMP and BACKFILL_## - as values to the --table option.

    Since there is no real - purpose to the list-tables command in the case of the Couchbase - Hadoop Connector, you are not recommended to use this argument.

    - Import all tables -

    Sqoop has a tool called import-all-tables. Couchbase does not have a - notion of tables.

    Since there is no real purpose to the - import-all-tables command in the case of the Couchbase Hadoop - Connector, it is not recommended you use this argument to Sqoop.

    -
    - Limitations - -

    While Couchbase provides many great features to import and export data from Couchbase to - Hadoop, there is some functionality that the connector does not implement in Sqoop:

    -
      -
    • -

      Querying: You cannot run queries on Couchbase. All tools that attempt to do this - will fail with a NotSupportedException.

      -
      • -
    • -

      list-databases tool: Even though Couchbase is a multitenant - system that allows for multiple buckets (which are analogous to databases), there is - no way of listing these buckets from Sqoop. The list of buckets is available - through the Couchbase Cluster web console.

      -
      • -
    • -

      eval-sql tool: Couchbase does not use SQL, so this tool is not - appropriate.

      -
      • - -
    • The Couchbase Hadoop Connector does not automatically handle some classes of - failure in a Couchbase cluster, and does not automatically handle changes to Couchbase cluster-topology, while the Sqoop task - is being run.
      • -
    - -
    +

    The Couchbase Hadoop Connector has reached End-of-Life (EOL) support. We recommend existing Hadoop integrations to migrate to a supported version of the Couchbase Kafka Connector. Additionally, the Couchbase Hadoop Connector is not compatible with Couchbase Server 5.x; because it relies on the TAP feed API which has been removed in Couchbase Server 5.x in favor of the DCP feed.

         diff --git a/content/connectors/intro.dita b/content/connectors/intro.dita index b563704e23..b49567678a 100644 --- a/content/connectors/intro.dita +++ b/content/connectors/intro.dita @@ -7,12 +7,12 @@

    Connectors are provided for:

      -
    • Elasticsearch
    • Hadoop
      • -
    • Kafka
      • +
    • Kafka
      • -
    • Spark
      • +
    • Spark
    • Talend
    diff --git a/content/connectors/kafka-3.2/quickstart.dita b/content/connectors/kafka-3.2/quickstart.dita index 477397b8e0..b727847bd5 100644 --- a/content/connectors/kafka-3.2/quickstart.dita +++ b/content/connectors/kafka-3.2/quickstart.dita @@ -7,8 +7,8 @@
    Download

    The connector is shipped as a zip archive, available for download: - kafka-connect-couchbase-3.2.1.zip

    + kafka-connect-couchbase-3.2.3.zip

    If you prefer to build the connectors from source, clone the GitHub repository and run mvn package to generate the connector archive @@ -80,7 +80,7 @@ kafka-server-start.sh $KAFKA_HOME/config/server.properties travel-sample (or whichever bucket you want to stream from). For connection.username and connection.password, supply the credentials of a Couchbase - user with read access to the bucket. If you have not yet created such a user, + user who has the "Data DCP Reader" role for the bucket. If you have not yet created such a user, now is a good time to read about Creating and Managing Users with the UI.

    diff --git a/content/connectors/kafka-3.2/release-notes.dita b/content/connectors/kafka-3.2/release-notes.dita index 7ab5d6daea..ec23da621a 100644 --- a/content/connectors/kafka-3.2/release-notes.dita +++ b/content/connectors/kafka-3.2/release-notes.dita @@ -4,6 +4,47 @@ Release Notes Release notes for the Kafka Connector. +
    + Couchbase Kafka Connector 3.2.3 GA (2018-02-20) + +

    Keepalive is now enabled on the Couchbase Server connection. This prevents the server from dropping + idle connections, and enables dead connection detection. Thanks to community member Patrik Nordebo (patrikn).

    +

    A new config key "couchbase.log_redaction" controls + whether sensitive info in connector log messages is tagged for redaction. + Options are "NONE", "PARTIAL", and "FULL".

    +

    A new source config key "couchbase.compression" can be used to enable + compression when reading from Couchbase Server 4.5 and later. + Options are "DISABLED", "ENABLED", and "FORCED".

    +

    The connector can now talk to Couchbase Server 5.5 and later over IPv6. + If Couchbase has been configured to run in IPv6 mode, set the environment variable + KAFKA_OPTS="-Dcom.couchbase.forceIPv4=false" + prior to running the connect worker. (The method for activating IPv6 may change + in a future release of the connector.)

    +

    Issues resolved in this release: +

      +
    • + KAFKAC-89: + [ENHANCEMENT] Enable NOOP for dead connection detection (Patrik Nordebo) +
      • +
    • + KAFKAC-82: + [FEATURE] Implement log redaction for Kafka Connector +
      • +
    • + KAFKAC-90: + [FEATURE] Source: Add config settings to enable compression +
      • +
    +

    +

    + + com.couchbase.client + kafka-connect-couchbase + 3.2.3 +]]> +

    +

    kafka-connect-couchbase-3.2.3.zip

    +
    Couchbase Kafka Connector 3.2.2 GA (19 December 2017) diff --git a/content/connectors/kafka-3.2/sink-configuration-options.dita b/content/connectors/kafka-3.2/sink-configuration-options.dita index 20bdb49770..63f797824b 100644 --- a/content/connectors/kafka-3.2/sink-configuration-options.dita +++ b/content/connectors/kafka-3.2/sink-configuration-options.dita @@ -113,6 +113,19 @@ PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
  • Importance: low
  • Default: "NONE"
    • +

    couchbase.log_redaction

    +

    Optionally tag sensitive values in the log output for later redaction. Possible values:

    +
      +
    • NONE - No redaction is performed.
      • +
    • PARTIAL - Only user data is redacted, system and metadata are not.
      • +
    • FULL - User, System and Metadata are all redacted.
      • +
    +
      +
    • Since: 3.2.3
      • +
    • Type: string
      • +
    • Importance: low
      • +
    • Default: "NONE"
      • +
    diff --git a/content/connectors/kafka-3.2/source-configuration-options.dita b/content/connectors/kafka-3.2/source-configuration-options.dita index b1a44a46f1..119b8d28aa 100644 --- a/content/connectors/kafka-3.2/source-configuration-options.dita +++ b/content/connectors/kafka-3.2/source-configuration-options.dita @@ -105,6 +105,36 @@ PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
  • Importance: low
  • Default: "SAVED_OFFSET_OR_BEGINNING"
    • +

    couchbase.log_redaction

    +

    Optionally tag sensitive values in the log output for later redaction. Possible values:

    +
      +
    • NONE - No redaction is performed.
      • +
    • PARTIAL - Only user data is redacted, system and metadata are not.
      • +
    • FULL - User, System and Metadata are all redacted.
      • +
    +
      +
    • Since: 3.2.3
      • +
    • Type: string
      • +
    • Importance: low
      • +
    • Default: "NONE"
      • +
    +

    couchbase.compression

    +

    To reduce bandwidth usage, Couchbase Server 4.5 and later can send documents to the connector in compressed form. + (Messages are always published to the Kafka topic in uncompressed form, regardless of this setting.) + Possible values:

    +
      +
    • DISABLED - No compression.
      • +
    • ENABLED - Couchbase Server decides whether to use compression on a per-document basis. For Couchbase 5.5 and later, + the document will be sent compressed if the server already has easy access to the compressed form. + For older server versions, this mode is equivalent to FORCED, and may increase server CPU load.
      • +
    • FORCED - Compression is used for every document, unless compressed size is greater than uncompressed size.
      • +
    +
      +
    • Since: 3.2.3
      • +
    • Type: string
      • +
    • Importance: low
      • +
    • Default: "DISABLED"
      • +
    diff --git a/content/connectors/kafka/_images/images/kafka3-kafka-connect.png b/content/connectors/kafka/_images/images/kafka3-kafka-connect.png new file mode 100644 index 0000000000..22cf94a4e8 Binary files /dev/null and b/content/connectors/kafka/_images/images/kafka3-kafka-connect.png differ diff --git a/content/connectors/kafka/_images/images/kafka3-run-source-connector.png b/content/connectors/kafka/_images/images/kafka3-run-source-connector.png new file mode 100644 index 0000000000..bd31722081 Binary files /dev/null and b/content/connectors/kafka/_images/images/kafka3-run-source-connector.png differ diff --git a/content/connectors/kafka/_images/images/kafka3-setup-source-connector.png b/content/connectors/kafka/_images/images/kafka3-setup-source-connector.png new file mode 100644 index 0000000000..0073ca1205 Binary files /dev/null and b/content/connectors/kafka/_images/images/kafka3-setup-source-connector.png differ diff --git a/content/connectors/kafka/kafka-dev-guide.ditamap b/content/connectors/kafka/kafka-dev-guide.ditamap new file mode 100644 index 0000000000..5e7728e756 --- /dev/null +++ b/content/connectors/kafka/kafka-dev-guide.ditamap @@ -0,0 +1,16 @@ + + + + Kafka Connector + + + + + + + + + + + + diff --git a/content/connectors/kafka/kafka-intro.dita b/content/connectors/kafka/kafka-intro.dita new file mode 100644 index 0000000000..2bcf00ea16 --- /dev/null +++ b/content/connectors/kafka/kafka-intro.dita @@ -0,0 +1,31 @@ + + + + Kafka Connector + The Kafka connector provides functionality to stream, filter, and transform events between + Couchbase Server and Kafka. + +

    The Couchbase sink connector allows you to import data from + Kafka + topics into Couchbase Server.

    + +

    The Couchbase source connector allows you to export data from Couchbase Server to Kafka topics.

    + +

    Data is loaded efficiently and reliably using Couchbase's internal replication protocol, DCP. Every change to or deletion + of the document generates a replication event, which is then sent to the configured Kafka topic. When copying data from a + bucket, the connector loads only new or modified documents.

    + +
    + Compatibility +

    Because the Kafka connector relies heavily on DCP, it is only compatible with Couchbase Server 3.0 and + later.

    +
    + +
    + Contributing +

    Couchbase welcomes community contributions to the Kafka connector. The Kafka Connector + source code is available on GitHub.

    +
    +     +     diff --git a/content/connectors/kafka/quickstart.html b/content/connectors/kafka/quickstart.html new file mode 100644 index 0000000000..ff2ee34a14 --- /dev/null +++ b/content/connectors/kafka/quickstart.html @@ -0,0 +1,689 @@ + + + + + + Couchbase Documentation + + + + + +
    +
    + +
    +
    + +
    + +
    +
    +

    This section shows how to add the connector to your application.

    +
    +
    +

    Download

    +
    +
    +

    The connector is shipped as a zip archive, available for download: +kafka-connect-couchbase-3.3.2.zip

    +
    +
    + + + + + +
    + + +A beta version with support for rollback mitigation is also available. +Try it and let us know about your experience: +kafka-connect-couchbase-3.4.0-beta.1.zip +
    +
    +
    +

    If you prefer to build the connectors from source, clone the +GitHub repository +and run mvn package to generate the connector archive (look for +kafka-connect-couchbase-<version>.zip in the target directory).

    +
    +
    +
    +
    +

    Getting Started

    +
    +
    +

    However you obtained the connector archive, go ahead and unzip it now. +The result should be a directory called +kafka-connect-couchbase-<version>. The rest of this guide will refer +to this directory as $KAFKA_CONNECT_COUCHBASE_HOME.

    +
    +
    +

    This guide assumes you have already installed Couchbase Server locally +and have loaded the sample bucket called travel-sample. (It’s fine if +you want to use a different bucket; neither the connector nor this guide +depend on the contents of the documents in the bucket.)

    +
    +
    +

    You’ll also need a local installation of Apache Kafka or Confluent +Platform Kafka.

    +
    +
    +
    +
    +

    Set Up Kafka

    +
    +
    +

    If you already have an installation of Kafka and know how to start the +servers, feel free to skip this section.

    +
    +
    +

    Still reading? Don’t worry, setting up a basic installation is pretty +easy. Download either Apache Kafka +or Confluent Platform Kafka. For +simplicity, this guide assumes you’re installing from a ZIP or TAR +archive, so steer clear of the deb/rpm packages for now.

    +
    +
    +

    Decompress the Apache Kafka or Confluent Platform archive and move the +resulting directory under ~/opt (or wherever you like to keep this +kind of software). The rest of this guide refers to the root of the +installation directory as $KAFKA_HOME or $CONFLUENT_HOME. Be aware +that some config files are in different relative locations depending on +whether you’re using Apache Kafka or Confluent Platform Kafka.

    +
    +
    +

    Make sure the Kafka command line tools are in your path:

    +
    +
    +
    +
    export PATH=<path-to-apache-kafka-or-confluent>/bin:$PATH
    +
    +
    +
    +
    +
    +

    Start the Kafka Servers

    +
    +
    +

    If you’re using Confluent Platform Kafka, start the servers by running +these commands, each in a separate terminal:

    +
    +
    +
    +
    zookeeper-server-start $CONFLUENT_HOME/etc/kafka/zookeeper.properties
+kafka-server-start $CONFLUENT_HOME/etc/kafka/server.properties
+schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties
    +
    +
    +
    +

    Note: Confluent 3.3.0 introduced a CLI tool that lets you start all the +servers with a single command. It also provides a more sophisticated way +to manage connectors than the technique described in this guide. Read +about it here.

    +
    +
    +

    If you’re not using Confluent, the commands are slightly different, but +the idea is the same. Start the servers by running these commands, each +in a separate terminal:

    +
    +
    +
    +
    zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
+kafka-server-start.sh $KAFKA_HOME/config/server.properties
    +
    +
    +
    +
    +
    +

    Configure the Source Connector

    +
    +
    +

    The Couchbase connector distribution includes sample config files. Look +inside $KAFKA_CONNECT_COUCHBASE_HOME/config and edit the +quickstart-couchbase-source.properties file.

    +
    +
    +

    Take a moment to peruse the configuration options specified here. Some +are standard +options available to all connectors. The rest are specific to the +Couchbase connector.

    +
    +
    +

    For this exercise, change the value of connection.bucket to +travel-sample (or whichever bucket you want to stream from). For +connection.username and connection.password, supply the credentials +of a Couchbase user who has the "Data DCP Reader" role for the bucket. +If you have not yet created such a user, now is a good time to read +about +Creating and +Managing Users with the UI.

    +
    +
    +

    Note: For Couchbase Server versions prior to 5.0, leave the username +blank. Set the password property to the bucket password, or leave it +blank if the bucket does not have a password. The sample buckets do not +have passwords.

    +
    +
    +

    You might also want to set use_snapshots to true, in which case the +source connector will buffer events until it receives a complete +snapshot before committing messages to the Kafka topic. It doesn’t +matter for this exercise; just be aware the option exists and defaults +to false.

    +
    +
    +
    +
    +

    Run the Source Connector

    +
    +
    +

    Kafka connectors can be run in +standalone or +distributed mode. For now let’s run the connector in standalone mode, +using the CLASSPATH environment variable to include the Couchbase +connector JAR in the class path.

    +
    +
    +

    For Confluent Platform Kafka:

    +
    +
    +
    +
    cd $KAFKA_CONNECT_COUCHBASE_HOME
+env CLASSPATH=./* \
+    connect-standalone $CONFLUENT_HOME/etc/schema-registry/connect-avro-standalone.properties \
+                       config/quickstart-couchbase-source.properties
    +
    +
    +
    +

    Or for Apache Kafka:

    +
    +
    +
    +
    cd $KAFKA_CONNECT_COUCHBASE_HOME
+env CLASSPATH=./* \
+    connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties \
+                       config/quickstart-couchbase-source.properties
    +
    +
    +
    +
    +
    +

    Alternatively, Run the Connector with Class Loader Isolation

    +
    +
    +

    Apache Kafka version 0.11.0 (and Confluent Platform 3.3.0) introduced a +mechanism for plugin class path isolation. To take advantage of this +feature, edit the connect worker config file (the connect-*.properties +file in the above commands). Modify the plugin.path property to +include the parent directory of kafka-connect-couchbase-<version>.jar.

    +
    +
    +

    Run the connector using the same commands as above, but omitting the +env CLASSPATH=./* prefix. Each Kafka Connect plugin will use a +separate class loader, removing the possibility of dependency conflicts.

    +
    +
    +
    +
    +

    Observe Messages Published by Couchbase Source Connector

    +
    +
    +

    The sample config file tells the source connector to publish to a topic +called test-default. Let’s use the Kafka command-line tools to spy on +the contents of the topic.

    +
    +
    +

    For Confluent Platform Kafka:

    +
    +
    +
    +
    kafka-avro-console-consumer --bootstrap-server localhost:9092 \
+                            --topic test-default --from-beginning
    +
    +
    +
    +

    Or for Apache Kafka:

    +
    +
    +
    +
    kafka-console-consumer.sh --bootstrap-server localhost:9092 \
+                          --topic test-default --from-beginning
    +
    +
    +
    +

    The expected output is a stream of Couchbase event notification +messages, at least one for each document in the bucket. The messages +include document metadata as well as content. The document content is +transferred as a byte array (encoded as Base64 if the connector is +configured to use JSON for message values).

    +
    +
    +

    Each message has an event field whose value indicates the type of +change represented by the message. The possible values are:

    +
    +
    +
      +
    • +

      mutation: A change to document content, including creation and +changes made via subdocument commands.

      +
      • +
    • +

      deletion: Removal or expiration of the document.

      +
      • +
    • +

      expiration: Reserved for document expiration (Couchbase Server does +not currently send this event type, but may in future versions).

      +
      • +
    +
    +
    +

    Once the consumer catches up to the current state of the bucket, try +adding a new document to the bucket +via the Couchbase Web Console. The consumer will print a notification +of type mutation. Now delete the document and watch for an event of +type deletion.

    +
    +
    +

    Perhaps it goes without saying, but all of the offset management and +fault tolerance features of Kafka Connect work with the Couchbase +connector. You can kill and restart the processes and they will pick up +where they left off.

    +
    +
    +

    The shape of the message payload is controlled by the +dcp.message.converter.class property of the connector config. By +default it is set to +com.couchbase.connect.kafka.converter.SchemaConverter, which formats +each notification into a structure that holds document metadata and +contents. For reference, the Avro schema for this payload format is +shown below:

    +
    +
    +
    +
    {
+  "type": "record",
+  "name": "DcpMessage",
+  "namespace": "com.couchbase",
+  "fields": [
+    {
+      "name": "event",
+      "type": "string"
+    },
+    {
+      "name": "partition",
+      "type": {
+        "type": "int",
+        "connect.type": "int16"
+      }
+    },
+    {
+      "name": "key",
+      "type": "string"
+    },
+    {
+      "name": "cas",
+      "type": "long"
+    },
+    {
+      "name": "bySeqno",
+      "type": "long"
+    },
+    {
+      "name": "revSeqno",
+      "type": "long"
+    },
+    {
+      "name": "expiration",
+      "type": [
+        "null",
+        "int"
+      ]
+    },
+    {
+      "name": "flags",
+      "type": [
+        "null",
+        "int"
+      ]
+    },
+    {
+      "name": "lockTime",
+      "type": [
+        "null",
+        "int"
+      ]
+    },
+    {
+      "name": "content",
+      "type": [
+        "null",
+        "bytes"
+      ]
+    }
+  ],
+  "connect.name": "com.couchbase.DcpMessage"
+}
    +
    +
    +
    +
    +
    +

    Couchbase Sink Connector

    +
    +
    +

    Now let’s talk about the sink connector, which reads messages from one +or more Kafka topics and writes them to Couchbase Server.

    +
    +
    +

    The sink connector will attempt to convert message values to JSON. If +the conversion fails, the connector will fall back to treating the value +as a String BLOB.

    +
    +
    +

    If the Kafka key is a primitive type, the connector will use it as the +document ID. If the Kafka key is absent or of complex type (array or +struct), the document ID will be generated as topic/partition/offset.

    +
    +
    +

    Alternatively, the document ID can come from the body of the Kafka +message. Provide a couchbase.document.id property whose value is a +JSON Pointer identifying the document ID node. If you want the connector +to remove this node before persisting the document to Couchbase, provide +a couchbase.remove.document.id property with value true. If the +connector fails to locate the document ID node, it will fall back to +using the Kafka key or topic/partition/offset as described above.

    +
    +
    +

    As of version 3.2.2, if the Kafka message body is null, the sink +connector will delete the Couchbase document whose ID matches the Kafka +message key.

    +
    +
    +
    +
    +

    Configure and Run the Sink Connector

    +
    +
    +

    In the $KAFKA_CONNECT_COUCHBASE_HOME/config directory there is a file +called quickstart-couchbase-sink.properties. Customize this file as +described in +Configure the +Source Connector, only now the bucket will receive messages and the +user must have write access to the bucket.

    +
    +
    +

    Note: Make sure to specify an existing bucket, otherwise the sink +connector will fail. You may wish to +create a new bucket to +receive the messages.

    +
    +
    +

    To run the sink connector, use the same command as described in Run the +Source Connector, but pass quickstart-couchbase-sink.properties as +the second argument to connect-standalone instead of +quickstart-couchbase-source.properties.

    +
    +
    +
    +
    +

    Send Test Messages

    +
    +
    +

    Now that the Couchbase Sink Connector is running, let’s give it some +messages to import:

    +
    +
    +
    +
    cd $KAFKA_CONNECT_COUCHBASE_HOME/examples/json-producer
+mvn compile exec:java
    +
    +
    +
    +

    The producer will send some messages and then terminate. If all goes +well, the messages will appear in the Couchbase bucket you specified in +the sink connector config..

    +
    +
    +

    If you wish to see how the Couchbase Sink Connector behaves in the +absence of message keys, modify the publishMessage method in the +example source code to set the message keys to null, then rerun the +producer.

    +
    +
    +

    Alternatively, if you want the Couchbase document ID to be the airport +code, edit quickstart-couchbase-sink.properties and set +couchbase.document.id=/airport, restart the sink connector, and run +the producer again.

    +
    +
    +

    Parent topic: Kafka Connector

    +
    +
    +

    Next topic: +Source Configuration Options

    +
    +
    +
    +
    + + + +
    +
    +
    +
    +

    This page was built using the Antora default UI.

    +

    The source code for this UI is licensed under the terms of the MPL-2.0 license.

    +
    + + + + + diff --git a/content/connectors/kafka/release-notes.html b/content/connectors/kafka/release-notes.html new file mode 100644 index 0000000000..d6e4732e94 --- /dev/null +++ b/content/connectors/kafka/release-notes.html @@ -0,0 +1,926 @@ + + + + + + Couchbase Documentation + + + + + +
    +
    + +
    +
    + +
    + +
    +
    +

    Release notes for the Kafka Connector.

    +
    +
    +

    Couchbase Kafka Connector 3.4.0-beta.1 (2018-07-06)

    +
    +
    +

    kafka-connect-couchbase-3.4.0-beta.1.zip

    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.4.0-beta.1</version>
+</dependency>
    +
    +
    +
    + + + + + +
    + + +Version 3.4.0-beta.1 has a bug that can cause data loss. When rollback mitigation is enabled, +some events might not be published to the Kafka topic. See KAFKAC-124 +for details. +
    +
    +
    +

    When database changes are rolled back due to failover of a Couchbase Server node, +the connector will now by default prevent the rolled-back changes from being published to the Kafka topic. +This is done by buffering changes until they have been persisted to all replicas +in the Couchbase cluster (and are unlikely to be rolled back). The connector polls the database at a configurable interval +to determine which changes have been persisted. This feature is enabled by default, +and may be disabled by setting the polling interval to 0.

    +
    +
    +

    The flow control buffer size is now configurable, with a default size of 128 megabytes. +This is an increase over the previous hard-coded value of 20 megabytes.

    +
    +
    +

    New configuration properties:

    +
    +
    +
      +
    • +

      couchbase.flow_control_buffer=128m

      +
      • +
    • +

      couchbase.persistence_polling_interval=100ms

      +
      • +
    +
    +
    +

    Issues resolved in this release:

    +
    +
    +
      +
    • +

      KAFKAC-116: +[ENHANCEMENT] Add support for rollback mitigation

      +
      • +
    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.3.2 GA (2018-07-02)

    +
    +
    +

    kafka-connect-couchbase-3.3.2.zip

    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.3.2</version>
+</dependency>
    +
    +
    +
    +

    The source connector now resumes streaming after a Couchbase Server rollback.

    +
    +
    +

    Issues resolved in this release:

    +
    +
    +
      +
    • +

      KAFKAC-72: +[BUGFIX] Connector stops streaming after rollback

      +
      • +
    • +

      KAFKAC-115: +[ENHANCEMENT] Upgrade to DCP client 0.18.0

      +
      • +
    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.3.1 GA (2018-06-19)

    +
    +
    +

    kafka-connect-couchbase-3.3.1.zip

    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.3.1</version>
+</dependency>
    +
    +
    +
    +

    The sink connector can now set expiration times for documents, +and can assign document IDs by composing values from multiple fields.

    +
    +
    +

    The source connector now continues streaming after a graceful failover.

    +
    +
    +

    Issues resolved in this release:

    +
    +
    +
      +
    • +

      KAFKAC-104: +[ENHANCEMENT] Support for setting TTL on items being written to Couchbase

      +
      • +
    • +

      KAFKAC-106: +[ENHANCEMENT] Support assigning document ID from multiple fields

      +
      • +
    • +

      KAFKAC-109: +[ENHANCEMENT] KAFKAC-109 Upgrade to Java client 2.5.9 and DCP client 0.17.0

      +
      • +
    • +

      JDCP-85: +[BUGFIX] Client stops receiving some events after failover

      +
      • +
    • +

      JDCP-94: +[BUGFIX] Should never pass null ChannelFlowController to event handlers

      +
      • +
    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.3.0 GA (2018-05-15)

    +
    +
    +

    kafka-connect-couchbase-3.3.0.zip

    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.3.0</version>
+</dependency>
    +
    +
    +
    +

    Compression is now enabled by default (requires Couchbase Server 5.5 or later).

    +
    +
    +

    IPv6 is now enabled by default. To disable IPv6 in a dual-stack environment, +set the new couchbase.forceIPv4 config property to true.

    +
    +
    +

    Resolved a regression that caused the connector to need a restart after +a Couchbase Server restart.

    +
    +
    +

    Couchbase password and SSL keystore password may be specified using via +KAFKA_COUCHBASE_PASSWORD and KAFKA_COUCHBASE_SSL_KEYSTORE_PASSWORD +environment variables.

    +
    +
    +

    The sink connector can now modify documents using the sub-document API +(many thanks to community member Didier Caron).

    +
    +
    +

    Issues resolved in this release:

    +
    +
    +
      +
    • +

      KAFKAC-97: +[BUGFIX] Kafka Connector needs a restart after Couchbase Server restart

      +
      • +
    • +

      KAFKAC-102: +[ENHANCEMENT] Enable compression by default

      +
      • +
    • +

      KAFKAC-100: +[ENHANCEMENT] Enable IPv6 by default, add first-class config option

      +
      • +
    • +

      KAFKAC-96: +[ENHANCEMENT] Support setting password via environment variable

      +
      • +
    • +

      KAFKAC-98: +[ENHANCEMENT] Sink: Support sub-document access (Didier Caron)

      +
      • +
    • +

      KAFKAC-103: +[ENHANCEMENT] Upgrade Couchbase dependencies

      +
      • +
    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.2.3 GA (2018-02-20)

    +
    +
    +

    Keepalive is now enabled on the Couchbase Server connection. This +prevents the server from dropping idle connections, and enables dead +connection detection. Thanks to community member Patrik Nordebo +(patrikn).

    +
    +
    +

    A new config key "couchbase.log_redaction" controls whether sensitive +info in connector log messages is tagged for redaction. Options are +"NONE", "PARTIAL", and "FULL".

    +
    +
    +

    A new source config key "couchbase.compression" can be used to enable +compression when reading from Couchbase Server 4.5 and later. Options +are "DISABLED", "ENABLED", and "FORCED".

    +
    +
    +

    Issues resolved in this release:

    +
    +
    +
      +
    • +

      KAFKAC-89: +[ENHANCEMENT] Enable NOOP for dead connection detection (Patrik Nordebo)

      +
      • +
    • +

      KAFKAC-82: [FEATURE] +Implement log redaction for Kafka Connector

      +
      • +
    • +

      KAFKAC-90: [FEATURE] +Source: Add config settings to enable compression

      +
      • +
    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.2.3</version>
+</dependency>
    +
    +
    +
    +

    kafka-connect-couchbase-3.2.3.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.2.2 GA (19 December 2017)

    +
    +
    +

    The source connector now does a better job of reporting abnormal +termination. Thanks to community member p_mx (tiny1990).

    +
    +
    +

    A new config key "couchbase.stream_from" lets you tell the source +connector when in Couchbase history to start streaming from. Options are +"BEGINNING", "NOW", "SAVED_OFFSET_OR_BEGINNING", and +"SAVED_OFFSET_OR_NOW".

    +
    +
    +

    When the sink connector receives a Kafka message with a null value, it +now deletes the Couchbase document whose ID matches the Kafka message +key. (Previous versions would terminate when a null value was +encountered.)

    +
    +
    +

    You can now specify durability requirements for the sink connector’s +write operations via two new config keys: +"couchbase.durability.persist_to" and +"couchbase.durability.replicate_to".

    +
    +
    +

    Issues resolved in this release:

    +
    +
    +
      +
    • +

      KAFKAC-84: [FEATURE] +Sink: Allow setting durability requirements for Couchbase writes

      +
      • +
    • +

      KAFKAC-85: [FEATURE] +Sink: Support deletion

      +
      • +
    • +

      KAFKAC-86: [FEATURE] +Source: Restart from a given state / offset

      +
      • +
    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.2.2</version>
+</dependency>
    +
    +
    +
    +

    kafka-connect-couchbase-3.2.2.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.2.1 GA (8 November 2017)

    +
    +
    +

    Fixes a regression in 3.2.0 that prevented the sink connector from +working.

    +
    +
    +

    Issues resolved in this release:

    +
    +
    +
      +
    • +

      KAFKAC-83: [BUGFIX] +Sink connector fails due to UnsupportedOperationException in +JsonBinaryTranscoder.newDocument.

      +
      • +
    • +

      KAFKAC-81: +[ENHANCEMENT] Document how to customize source handler

      +
      • +
    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.2.1</version>
+</dependency>
    +
    +
    +
    +

    kafka-connect-couchbase-3.2.1.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.2.0 GA (17 October 2017)

    +
    +
    +

    The sink connector is now officially supported.

    +
    +
    +

    Multiple source connector instances reading from the same Couchbase +bucket can now manage their state independently. To enable this feature, +assign a unique name to each connector and set the new config property +compat.connector_name_in_offsets to true.

    +
    +
    +

    The source connector has a new, more flexible SourceHandler extension +point intended to replace the Converter interface. By providing a +custom SourceHandler, a developer can filter events, route messages to +a topic other than the configured default topic, and control the format +of the Kafka message.

    +
    +
    +

    The default Kafka message format now includes bucket and vBucketUuid +fields, which may be used along with the partition (vBucketID) and +bySeqno fields to construct a Couchbase MutationToken.

    +
    +
    +

    The sink connector now allows the Couchbase document ID to come from a +field of the document. This feature is controlled by two new config +properties, couchbase.document.id and couchbase.remove.document.id.

    +
    +
    +

    The Couchbase client libraries are upgraded to the latest versions, with +performance enhancements and other improvements for Couchbase Server +5.0.

    +
    +
    +

    The following classes are scheduled for removal in version 4.0.0 of the +connector:

    +
    +
    +
      +
    • +

      com.couchbase.connect.kafka.converter.Converter - Deprecated in +favor of SourceHandler.

      +
      • +
    • +

      com.couchbase.connect.kafka.converter.SchemaConverter - Deprecated +in favor of DefaultSchemaSourceHandler.

      +
      • +
    +
    +
    +

    Issues resolved in this release:

    +
    +
    +
      +
    • +

      KAFKAC-69: [FEATURE] +Allow Source connector to split DCP stream and write into separate +topics.

      +
      • +
    • +

      KAFKAC-70: [FEATURE] +Allow using connector name in offset storage namespace

      +
      • +
    • +

      KAFKAC-77: [FEATURE] +Allow setting document ID from message field.

      +
      • +
    • +

      KAFKAC-78: +[ENHANCEMENT] Upgrade Couchbase java-client to version 2.5.1, dcp-client +to version 0.12.0

      +
      • +
    • +

      KAFKAC-79: +[ENHANCEMENT] Use custom doc transcoder to reduce memory copies.

      +
      • +
    • +

      KAFKAC-80: [FEATURE] +MutationToken Enabled in Kafka Connector to N1QL at plus.

      +
      • +
    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.2.0</version>
+</dependency>
    +
    +
    +
    +

    kafka-connect-couchbase-3.2.0.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.1.3 GA (31 May 2017)

    +
    +
    +

    Version 3.1.3 is maintenance release.

    +
    +
    +
      +
    • +

      KAFKAC-71: Support for +RBAC credentials on Couchbase Server 5+.

      +
      • +
    • +

      Java DCP Client updated to 0.10.0.

      +
      • +
    +
    +
    +
    +
    <dependency>
+    <groupId>com.couchbase.client</groupId>
+    <artifactId>kafka-connect-couchbase</artifactId>
+    <version>3.1.3</version>
+</dependency>
    +
    +
    +
    +

    kafka-connect-couchbase-3.1.3.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.1.2 GA (14 March 2017)

    +
    +
    +

    Version 3.1.2 is maintenance release..

    +
    +
    +
      +
    • +

      KAFKAC-66: On +backfilling from large bucket, it is possible to get OOM when internal +queue is not drained quickly enough to relay the data into Kafka.

      +
      • +
    +
    +
    +

    kafka-connect-couchbase-3.1.2.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.1.1 GA (21 February 2017)

    +
    +
    +

    Version 3.1.1 is maintenance release. It contains fixes for resuming DCP +streams after restart.

    +
    +
    +
      +
    • +

      KAFKAC-56: Session +state might be left partially initialized, which leads to rolling back +to sequence number zero (0) and starting from the beginning (duplicating +events in Kafka topic).

      +
      • +
    +
    +
    +

    kafka-connect-couchbase-3.1.1.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.1.0 GA (03 January 2017)

    +
    +
    +

    Version 3.1.0 is GA release.

    +
    +
    + +
    +
    +

    kafka-connect-couchbase-3.1.0.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.0.0 GA (14 December 2016)

    +
    +
    +

    Version 3.0.0 is GA release. It brings documentation update.

    +
    +
    +

    kafka-connect-couchbase-3.0.0.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.0.0 BETA (22 November 2016)

    +
    +
    +

    Version 3.0.0-BETA is pre-release version of the 3.0.0. It brings +documentation update, feature enhancements and bug fixes

    +
    +
    +
      +
    • +

      KAFKAC-52: Support for +SSL connections

      +
      • +
    • +

      Update dependencies: dcp-client to 0.7.0, and confluent libraries up +to versions shipped with 3.1.1

      +
      • +
    • +

      Cleanup various configuration workarounds for platform 3.0

      +
      • +
    +
    +
    +

    kafka-connect-couchbase-3.0.0-BETA.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.0.0 DP4 (5 November 2016)

    +
    +
    +

    Version 3.0.0-DP4 is the fourth developer preview of the 3.0.x series.

    +
    +
    +
      +
    • +

      KAFKAC-54: Create +example of using in Kafka Stream to process events from Couchbase

      +
      • +
    • +

      Rename internal classes, and make configuration more consistent with +other connectors (e.g. instead of timeout_ms, use timeout.ms)

      +
      • +
    • +

      Allow to override internal convertor into SourceRecord, and allow to +inject Filter class to skip events before writing into Kafka

      +
      • +
    +
    +
    +

    kafka-connect-couchbase-3.0.0-DP4.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.0.0 DP3 (20 October 2016)

    +
    +
    +

    Version 3.0.0-DP3 is the third developer preview of the 3.0.x series. It +implements new features and also includes bug fixes to previous release.

    +
    +
    +
      +
    • +

      KAFKAC-50: Allow to +buffer DCP snapshots for consistent writes.

      +
      • +
    • +

      KAFKAC-51: Specify key +for SourceRecord. Allows to use multiple Kafka partitions.

      +
      • +
    • +

      KAFKAC-53: Node-aware +distribution of partitions for Tasks. Reduces amount of resources +allocated on the server.

      +
      • +
    +
    +
    +

    kafka-connect-couchbase-3.0.0-DP3.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.0.0 DP2 (24 September 2016)

    +
    +
    +

    Version 3.0.0-DP2 is the second developer preview of the 3.0.x series. +It improves configuration. And now can maintain replication state, which +allow to resume transmission.

    +
    +
    +

    kafka-connect-couchbase-3.0.0-DP2.zip

    +
    +
    +
    +
    +

    Couchbase Kafka Connector 3.0.0 DP1 (6 September 2016)

    +
    +
    +

    Version 3.0.0-DP1 is the first developer preview of the 3.0.x series.

    +
    +
    +

    kafka-connect-couchbase-3.0.0-DP1.zip

    +
    +
    +

    Parent topic: Kafka Connector

    +
    +
    +

    Previous topic: +Couchbase Sample with Kafka Streams

    +
    +
    +
    +
    + + + +
    +
    +
    +
    +

    This page was built using the Antora default UI.

    +

    The source code for this UI is licensed under the terms of the MPL-2.0 license.

    +
    + + + + + diff --git a/content/connectors/kafka/sink-configuration-options.html b/content/connectors/kafka/sink-configuration-options.html new file mode 100644 index 0000000000..d6b833d7ac --- /dev/null +++ b/content/connectors/kafka/sink-configuration-options.html @@ -0,0 +1,672 @@ + + + + + + Couchbase Documentation + + + + + +
    +
    + +
    +
    + +
    + +
    +
    +

    Reference of the source connector options.

    +
    +
    +

    Database

    +
    +
    +

    connection.cluster_address

    +
    +

    Couchbase Cluster addresses to listen (use comma to specify several).

    +
    +
    +
      +
    • +

      Type: list

      +
      • +
    • +

      Importance: high

      +
      • +
    +
    +
    +
    +

    connection.bucket

    +
    +

    Couchbase bucket name.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: high

      +
      • +
    +
    +
    +
    +

    connection.username

    +
    +

    Couchbase username for Couchbase Server 5.0 and later.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: high

      +
      • +
    • +

      Default: the bucket name specified by connection.bucket

      +
      • +
    +
    +
    +
    +

    connection.password

    +
    +

    Couchbase password. For Couchbase Server versions prior to 5.0 this is +the bucket password. For versions 5.0 and later, this is the password +associated with connection.username.

    +
    +
    +
      +
    • +

      Type: password

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: ""

      +
      • +
    +
    +
    +
    +

    connection.timeout.ms

    +
    +

    Connection timeout in milliseconds.

    +
    +
    +
      +
    • +

      Type: long

      +
      • +
    • +

      Importance: low

      +
      • +
    +
    +
    +
    +

    connection.ssl.enabled

    +
    +

    Use SSL to connect to Couchbase. This feature only available in +Couchbase Enterprise.

    +
    +
    +
      +
    • +

      Type: boolean

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: false

      +
      • +
    • +

      Dependents: connection.ssl.keystore.location, +connection.ssl.keystore.password

      +
      • +
    +
    +
    +
    +

    connection.ssl.keystore.location

    +
    +

    The location of the key store file.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    +
    +
    +
    +

    connection.ssl.keystore.password

    +
    +

    The password of the private key in the key store file.

    +
    +
    +
      +
    • +

      Type: password

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: ""

      +
      • +
    +
    +
    +
    +

    couchbase.document.id

    +
    +

    Optionally set the Couchbase document ID from fields of the message body. +The value is a format string with a placeholder for each field to include +in the ID. A placeholder looks like ${/path/to/field} where the path +is specified as a JSON pointer.

    +
    +
    +

    For backwards compatibility, a format string without placeholders +is interpreted as a single JSON pointer.

    +
    +
    +

    The examples below assume the following sample document content:

    +
    +
    +
    +
    {
+  "id": 123,
+  "metadata": {
+    "type": "widget"
+  }
+}
    +
    +
    +
    +
      +
    • +

      Example values:

      +
      +
        +
      • +

        ${/id} → 123

        +
        • +
      • +

        invoice::${/id} → invoice::123

        +
        • +
      • +

        ${/metadata/type}::${/id} → widget::123

        +
        • +
      • +

        /id → 123 (deprecated, use ${/id} instead)

        +
        • +
      +
      +
      • +
    • +

      Since: 3.3.1 (Uncommitted; API may change in a patch release)

      +
      • +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: ""

      +
      • +
    • +

      Dependents: couchbase.remove.document.id

      +
      • +
    +
    +
    +
    +

    couchbase.remove.document.id

    +
    +

    Controls whether the ID field is removed from the document when +couchbase.document.id is specified.

    +
    +
    +
      +
    • +

      Type: boolean

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: false

      +
      • +
    +
    +
    +
    +

    couchbase.durability.persist_to

    +
    +

    Optionally specify Couchbase persistence requirements for a write to be +considered successful. If the requested requirements cannot be met (due +to Couchbase rebalance or failover, for instance) the connector will +terminate.

    +
    +
    +
      +
    • +

      Possible values:

      +
      +
        +
      • +

        NONE - Do not require any disk persistence.

        +
        • +
      • +

        MASTER - Require disk persistence to the master node of the document +only.

        +
        • +
      • +

        ONE - Require disk persistence of one node (master or replica).

        +
        • +
      • +

        TWO - Require disk persistence of two nodes (master or replica).

        +
        • +
      • +

        THREE - Require disk persistence of three nodes (master or replica).

        +
        • +
      • +

        FOUR - Require disk persistence of four nodes (master + three +replicas).

        +
        • +
      +
      +
      • +
    • +

      Since: 3.2.2

      +
      • +
    • +

      Type: boolean

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "NONE"

      +
      • +
    +
    +
    +
    +

    couchbase.durability.replicate_to

    +
    +

    Optionally specify Couchbase replication requirements for a write to be +considered successful. If the requested requirements cannot be met (due +to Couchbase rebalance or failover, for instance) the connector will +terminate.

    +
    +
    +
      +
    • +

      Possible values:

      +
      +
        +
      • +

        NONE - Do not require any replication.

        +
        • +
      • +

        ONE - Require replication to one replica.

        +
        • +
      • +

        TWO - Require replication to two replicas.

        +
        • +
      • +

        THREE - Require replication to three replicas.

        +
        • +
      +
      +
      • +
    • +

      Since: 3.2.2

      +
      • +
    • +

      Type: boolean

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "NONE"

      +
      • +
    +
    +
    +
    +

    couchbase.log_redaction

    +
    +

    Optionally tag sensitive values in the log output for later redaction.

    +
    +
    +
      +
    • +

      Possible values:

      +
      +
        +
      • +

        NONE - No redaction is performed.

        +
        • +
      • +

        PARTIAL - Only user data is redacted, system and metadata are not.

        +
        • +
      • +

        FULL - User, System and Metadata are all redacted.

        +
        • +
      +
      +
      • +
    • +

      Since: 3.2.3

      +
      • +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "NONE"

      +
      • +
    +
    +
    +
    +

    couchbase.forceIPv4

    +
    +

    In a network environment that supports both IPv4 and IPv6, setting this property +to true will force the use of IPv4 when resolving Couchbase Server hostnames.

    +
    +
    +
      +
    • +

      Since: 3.3.0

      +
      • +
    • +

      Type: boolean

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: false

      +
      • +
    +
    +
    +
    +

    couchbase.document.expiration

    +
    +

    Optionally specify a time-to-live for documents written to Couchbase. +If present, the value must be an integer followed by a time unit. +(s = seconds, m = minutes, h = hours, d = days). Example value: 30m.

    +
    +
    +
      +
    • +

      Since: 3.3.1 (Uncommitted; API may change in a patch release)

      +
      • +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "" (documents will not expire)

      +
      • +
    +
    +
    +

    Parent topic: Kafka Connector

    +
    +
    +

    Previous topic: +Source Configuration Options

    +
    +
    +

    Next topic: +Couchbase Sample with Kafka Streams

    +
    +
    +
    +
    +
    + + + +
    +
    +
    +
    +

    This page was built using the Antora default UI.

    +

    The source code for this UI is licensed under the terms of the MPL-2.0 license.

    +
    + + + + + diff --git a/content/connectors/kafka/source-configuration-options.html b/content/connectors/kafka/source-configuration-options.html new file mode 100644 index 0000000000..d833bd83ab --- /dev/null +++ b/content/connectors/kafka/source-configuration-options.html @@ -0,0 +1,731 @@ + + + + + + Couchbase Documentation + + + + + +
    +
    + +
    +
    + +
    + +
    +
    +

    Reference of the source connector options.

    +
    +
    +

    Database

    +
    +
    +

    connection.cluster_address

    +
    +

    Couchbase Cluster addresses to listen (use comma to specify several).

    +
    +
    +
      +
    • +

      Type: list

      +
      • +
    • +

      Importance: high

      +
      • +
    +
    +
    +
    +

    connection.bucket

    +
    +

    Couchbase bucket name.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: high

      +
      • +
    +
    +
    +
    +

    connection.username

    +
    +

    Couchbase username for Couchbase Server 5.0 and later.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: high

      +
      • +
    • +

      Default: the bucket name specified by connection.bucket

      +
      • +
    +
    +
    +
    +

    connection.password

    +
    +

    Couchbase password. For Couchbase Server versions prior to 5.0 this is +the bucket password. For versions 5.0 and later, this is the password +associated with connection.username.

    +
    +
    +
      +
    • +

      Type: password

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: ""

      +
      • +
    +
    +
    +
    +

    connection.timeout.ms

    +
    +

    Connection timeout in milliseconds.

    +
    +
    +
      +
    • +

      Type: long

      +
      • +
    • +

      Importance: low

      +
      • +
    +
    +
    +
    +

    connection.ssl.enabled

    +
    +

    Use SSL to connect to Couchbase. This feature only available in +Couchbase Enterprise.

    +
    +
    +
      +
    • +

      Type: boolean

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: false

      +
      • +
    • +

      Dependents: connection.ssl.keystore.location, +connection.ssl.keystore.password

      +
      • +
    +
    +
    +
    +

    connection.ssl.keystore.location

    +
    +

    The location of the key store file.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    +
    +
    +
    +

    connection.ssl.keystore.password

    +
    +

    The password of the private key in the key store file.

    +
    +
    +
      +
    • +

      Type: password

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: ""

      +
      • +
    +
    +
    +
    +
    +
    +

    Connector

    +
    +
    +

    topic.name

    +
    +

    Name of the Kafka topic to publish data to.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: high

      +
      • +
    +
    +
    +
    +

    use_snapshots

    +
    +

    If true, it will only commit into Kafka when full snapshot from +Couchbase received.

    +
    +
    +
      +
    • +

      Type: boolean

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: false

      +
      • +
    +
    +
    +
    +

    dcp.message.converter.class

    +
    +

    The class name of the message converter to use.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: +"com.couchbase.connect.kafka.handler.source.DefaultSchemaSourceHandler"

      +
      • +
    +
    +
    +
    +

    event.filter.class

    +
    +

    The class name of the event filter to use.

    +
    +
    +
      +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "com.couchbase.connect.kafka.filter.AllPassFilter"

      +
      • +
    +
    +
    +
    +

    couchbase.stream_from

    +
    +

    The point in Couchbase history to start streaming from.

    +
    +
    +
      +
    • +

      Possible values:

      +
      +
        +
      • +

        SAVED_OFFSET_OR_BEGINNING - Restart from saved state, or if none, +restart from oldest available mutation in Couchbase

        +
        • +
      • +

        SAVED_OFFSET_OR_NOW - Restart from saved state, or if none, restart +from current Couchbase state

        +
        • +
      • +

        BEGINNING - Restart from oldest available mutation in Couchbase +(ignore any potential saved state)

        +
        • +
      • +

        NOW - Restart from current Couchbase state (ignore any potential saved +state)

        +
        • +
      +
      +
      • +
    • +

      Since: 3.2.2

      +
      • +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "SAVED_OFFSET_OR_BEGINNING"

      +
      • +
    +
    +
    +
    +

    couchbase.log_redaction

    +
    +

    Optionally tag sensitive values in the log output for later redaction.

    +
    +
    +
      +
    • +

      Possible values:

      +
      +
        +
      • +

        NONE - No redaction is performed.

        +
        • +
      • +

        PARTIAL - Only user data is redacted, system and metadata are not.

        +
        • +
      • +

        FULL - User, System and Metadata are all redacted.

        +
        • +
      +
      +
      • +
    • +

      Since: 3.2.3

      +
      • +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "NONE"

      +
      • +
    +
    +
    +
    +

    couchbase.compression

    +
    +

    To reduce bandwidth usage, Couchbase Server 5.5 and later can send +documents to the connector in compressed form. (Messages are always +published to the Kafka topic in uncompressed form, regardless of this +setting.) If the requested mode is not supported by your version of +Couchbase Server, compression will be disabled.

    +
    +
    +
      +
    • +

      Possible values:

      +
      +
        +
      • +

        ENABLED - (default) Couchbase Server decides whether to use compression +on a per-document basis, depending on whether the compressed form of the +document is readily available. Select this mode to prioritize Couchbase Server +performance and reduced bandwidth usage (recommended). Requires Couchbase Server 5.5 or later.

        +
        • +
      • +

        DISABLED - No compression. Select this mode to prioritize reduced CPU load +for the Kafka connector.

        +
        • +
      • +

        FORCED - Compression is used for every document, unless compressed +size is greater than uncompressed size. Select this mode to prioritize bandwidth +usage reduction above all else. Requires Couchbase Server 5.5 or later.

        +
        • +
      +
      +
      • +
    • +

      Since: 3.3.0

      +
      • +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "ENABLED"

      +
      • +
    +
    +
    +
    +

    couchbase.forceIPv4

    +
    +

    In a network environment that supports both IPv4 and IPv6, setting this property +to true will force the use of IPv4 when resolving Couchbase Server hostnames.

    +
    +
    +
      +
    • +

      Since: 3.3.0

      +
      • +
    • +

      Type: boolean

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: false

      +
      • +
    +
    +
    +
    +

    couchbase.persistence_polling_interval

    +
    +

    The frequency at which the connector will poll Couchbase Server to determine +which database changes are unlikely to be rolled back. A value of 0 +disables polling and causes changes to be published to the Kafka topic as soon as they are received.

    +
    +
    + + + + + +
    + + +If polling is disabled, when rollbacks occur you are more likely to end up with events +in the Kafka topic that do not match the actual database state, because +they are from an "alternate timeline" in Couchbase Server’s history. +
    +
    +
    +

    The longer the polling interval, the larger the flow control buffer required +in order to maintain steady throughput, since events are buffered +until persistence is confirmed.

    +
    +
    +

    If present, the value must be 0 or an integer followed by a time unit: +(ms = milliseconds, s = seconds)

    +
    +
    +
      +
    • +

      Since: 3.4.0-beta.1 (Uncommitted; API may change in a patch release)

      +
      • +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "100ms"

      +
      • +
    +
    +
    +
    +

    couchbase.flow_control_buffer

    +
    +

    The amount of heap space to reserve for the flow control buffer. +This is the amount of data Couchbase Server will push to the connector +before awaiting acknowledgement that the data has been published to the Kafka topic.

    +
    +
    +

    If present, the value must be an integer followed by a storage size unit: +(b = bytes, k = kilobytes, m = megabytes, g = gigabytes)

    +
    +
    + + + + + +
    + + +This value must be smaller than the connect worker’s heap size. +To allocate more memory to the connect worker process, +set the KAFKA_HEAP_OPTS environment variable before starting the worker. +For example: export KAFKA_HEAP_OPTS=-Xmx1024M +
    +
    +
    +
      +
    • +

      Since: 3.4.0-beta.1 (Uncommitted; API may change in a patch release)

      +
      • +
    • +

      Type: string

      +
      • +
    • +

      Importance: low

      +
      • +
    • +

      Default: "128m"

      +
      • +
    +
    +
    +

    Parent topic: Kafka Connector

    +
    +
    +

    Previous topic: +Quickstart

    +
    +
    +

    Next topic: +Sink Configuration Options

    +
    +
    +
    +
    +
    + + + +
    +
    +
    +
    +

    This page was built using the Antora default UI.

    +

    The source code for this UI is licensed under the terms of the MPL-2.0 license.

    +
    + + + + + diff --git a/content/connectors/kafka/streams-sample.html b/content/connectors/kafka/streams-sample.html new file mode 100644 index 0000000000..ddf20a93c0 --- /dev/null +++ b/content/connectors/kafka/streams-sample.html @@ -0,0 +1,599 @@ + + + + + + Couchbase Documentation + + + + + +
    +
    + +
    +
    + +
    + +
    +
    +

    Sample application which uses Couchbase connector with Kafka Streams.

    +
    +
    +

    Prerequisites

    +
    +
    +

    This example demonstrates how to build a data pipeline using Kafka to +move data from Couchbase Server to a MySQL database. It assumes a +Couchbase Server instance with the beer-sample bucket deployed on +localhost and a MySQL server accessible on its default port (3306). +MySQL should also have a beer_sample_sql database. The following +snippet describes the schema of the database:

    +
    +
    +
    +
    DROP DATABASE IF EXISTS beer_sample_sql;
+CREATE DATABASE beer_sample_sql CHARACTER SET utf8 COLLATE utf8_general_ci;
+USE beer_sample_sql;
+CREATE TABLE breweries (
+   id VARCHAR(256) NOT NULL,
+   name VARCHAR(256),
+   description TEXT,
+   country VARCHAR(256),
+   city VARCHAR(256),
+   state VARCHAR(256),
+   phone VARCHAR(40),
+   updated_at DATETIME,
+   PRIMARY KEY (id)
+);
+CREATE TABLE beers (
+   id VARCHAR(256) NOT NULL,
+   brewery_id VARCHAR(256) NOT NULL,
+   name VARCHAR(256),
+   category VARCHAR(256),
+   style VARCHAR(256),
+   description TEXT,
+   abv DECIMAL(10,2),
+   ibu DECIMAL(10,2),
+   updated_at DATETIME,
+   PRIMARY KEY (id)
+);
    +
    +
    +
    +

    This example is built on top of the +Confluent Platform +which also installed on localhost, along with the Couchbase connector. +We will use the Confluent +Control +Center to configure the link, so make sure this service also is +running. The commands below can be used to start all dependencies:

    +
    +
    +
    +
    $ service couchbase-server start
+$ service mysql-server start
+
+# For RPM/DEB based Confluent Platform deployments the paths might be absolute.
+$ ./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties &
+$ ./bin/kafka-server-start ./etc/kafka/server.properties &
+$ ./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties &
+
+# Run connect framework in distributed mode
+$ ./bin/connect-distributed $CONNECTOR_DIST/config/connect-distributed.properties
+
+$ ./bin/control-center-start etc/confluent-control-center/control-center.properties
    +
    +
    +
    +

    Note that for the connect-distributed script we use the configuration +from the couchbase connector. You can use a stock configuration too, but +make sure that it will use Avro convertors and configure interceptors +for monitoring:

    +
    +
    +
    +
    key.converter=io.confluent.connect.avro.AvroConverter
+key.converter.schema.registry.url=http://localhost:8081
+value.converter=io.confluent.connect.avro.AvroConverter
+value.converter.schema.registry.url=http://localhost:8081
+consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
+producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
    +
    +
    +
    +
    +
    +

    Code Overview

    +
    +
    +

    The full code of this sample is accessible in the connector repository +at +src/test/java/examples/KafkaStreamsDemo.java.

    +
    +
    +
    +
    try {
+    Class.forName("com.mysql.jdbc.Driver");
+} catch (ClassNotFoundException e) {
+    System.err.println("Failed to load MySQL JDBC driver");
+}
+Connection connection = DriverManager
+        .getConnection("jdbc:mysql://localhost:3306/beer_sample_sql", "root", "secret");
+final PreparedStatement insertBrewery = connection.prepareStatement(
+        "INSERT INTO breweries (id, name, description, country, city, state, phone, updated_at)" +
+                " VALUES (?, ?, ?, ?, ?, ?, ?, ?)" +
+                " ON DUPLICATE KEY UPDATE" +
+                " name=VALUES(name), description=VALUES(description), country=VALUES(country)," +
+                " country=VALUES(country), city=VALUES(city), state=VALUES(state)," +
+                " phone=VALUES(phone), updated_at=VALUES(updated_at)");
+final PreparedStatement insertBeer = connection.prepareStatement(
+        "INSERT INTO beers (id, brewery_id, name, description, category, style, abv, ibu, updated_at)" +
+                " VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)" +
+                " ON DUPLICATE KEY UPDATE" +
+                " brewery_id=VALUES(brewery_id), name=VALUES(name), description=VALUES(description)," +
+                " category=VALUES(category), style=VALUES(style), abv=VALUES(abv)," +
+                " ibu=VALUES(ibu), updated_at=VALUES(updated_at)");
    +
    +
    +
    +

    The main function of the KafkaStreamsDemo class starts with loading +the MySQL driver, establishing connection and preparing insert +statements for both kinds of the documents: brewery and beer.

    +
    +
    +
    +
    String schemaRegistryUrl = "http://localhost:8081";
+Properties props = new Properties();
+props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-test");
+props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+props.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, "localhost:2181");
+props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
+props.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, KeyAvroSerde.class);
+props.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, ValueAvroSerde.class);
+props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    +
    +
    +
    +

    The next block supplies the Streams configuration. Along with endpoints, +it sets serializers and deserializers for keys and values, which appear +in the Kafka topics. These values are written by the Couchbase +connector. Here we use simple classes +src/test/java/examples/serde/KeyAvroSerde.java +and +src/test/java/examples/serde/ValueAvroSerde.java, +which do not make assumptions about the document body, but the real +application might implement serdes, working with more specific classes.

    +
    +
    +
    +
    KStreamBuilder builder = new KStreamBuilder();
+
+KStream<String, GenericRecord> source = builder
+        .stream("streaming-topic-beer-sample");
    +
    +
    +
    +

    We start constructing the source stream by pulling data from the Kafka +topic streaming-topic-beer-sample.

    +
    +
    +
    +
    KStream<String, JsonNode>[] documents = source
+        .mapValues(new ValueMapper<GenericRecord, JsonNode>() {
+            @Override
+            public JsonNode apply(GenericRecord value) {
+                ByteBuffer buf = (ByteBuffer) value.get("content");
+                try {
+                    JsonNode doc = MAPPER.readTree(buf.array());
+                    return doc;
+                } catch (IOException e) {
+                    return null;
+                }
+            }
+        })
+        .branch(
+                new Predicate<String, JsonNode>() {
+                    @Override
+                    public boolean test(String key, JsonNode value) {
+                        return "beer".equals(value.get("type").asText()) &&
+                                value.has("brewery_id") &&
+                                value.has("name") &&
+                                value.has("description") &&
+                                value.has("category") &&
+                                value.has("style") &&
+                                value.has("abv") &&
+                                value.has("ibu") &&
+                                value.has("updated");
+                    }
+                },
+                new Predicate<String, JsonNode>() {
+                    @Override
+                    public boolean test(String key, JsonNode value) {
+                        return "brewery".equals(value.get("type").asText()) &&
+                                value.has("name") &&
+                                value.has("description") &&
+                                value.has("country") &&
+                                value.has("city") &&
+                                value.has("state") &&
+                                value.has("phone") &&
+                                value.has("updated");
+                    }
+                }
+        );
    +
    +
    +
    +

    The first step in our pipeline would be to extract content from the +Couchbase event and deserialize it as JSON, as Couchbase operates with +JSON documents normally, and in beer-sample bucket in particular. With +branch operator, we split stream into two by the document type, and in +the same type we filter out documents that don’t have all the fields we +want to insert into the MySQL database.

    +
    +
    +
    +
    documents[0].foreach(new ForeachAction<String, JsonNode>() {
+    @Override
+    public void apply(String key, JsonNode value) {
+        try {
+            insertBeer.setString(1, key);
+            insertBeer.setString(2, value.get("brewery_id").asText());
+            insertBeer.setString(3, value.get("name").asText());
+            insertBeer.setString(4, value.get("description").asText());
+            insertBeer.setString(5, value.get("category").asText());
+            insertBeer.setString(6, value.get("style").asText());
+            insertBeer.setBigDecimal(7, new BigDecimal(value.get("abv").asText()));
+            insertBeer.setBigDecimal(8, new BigDecimal(value.get("ibu").asText()));
+            insertBeer.setDate(9, new Date(DATE_FORMAT.parse(value.get("updated").asText()).getTime()));
+            insertBeer.execute();
+        } catch (SQLException e) {
+            System.err.println("Failed to insert record: " + key + ". " + e);
+        } catch (ParseException e) {
+            System.err.println("Failed to insert record: " + key + ". " + e);
+        }
+    }
+});
+documents[1].foreach(new ForeachAction<String, JsonNode>() {
+    @Override
+    public void apply(String key, JsonNode value) {
+        try {
+            insertBrewery.setString(1, key);
+            insertBrewery.setString(2, value.get("name").asText());
+            insertBrewery.setString(3, value.get("description").asText());
+            insertBrewery.setString(4, value.get("country").asText());
+            insertBrewery.setString(5, value.get("city").asText());
+            insertBrewery.setString(6, value.get("state").asText());
+            insertBrewery.setString(7, value.get("phone").asText());
+            insertBrewery.setDate(8, new Date(DATE_FORMAT.parse(value.get("updated").asText()).getTime()));
+            insertBrewery.execute();
+        } catch (SQLException e) {
+            System.err.println("Failed to insert record: " + key + ". " + e);
+        } catch (ParseException e) {
+            System.err.println("Failed to insert record: " + key + ". " + e);
+        }
+    }
+});
    +
    +
    +
    +

    Once the documents are extracted and filtered we are ready to insert +them into the MySQL database using statements prepared earlier. Note +that inserted records are using the document ID from Couchbase, which +means that records will be updated in place automatically without +creating duplicates. This example does not handle document deletions or +expiration, but it won’t be complex to do with an additional stream that +executes DELETE statements

    +
    +
    +
    +
    final KafkaStreams streams = new KafkaStreams(builder, props);
+streams.start();
+Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
+    @Override
+    public void run() {
+        streams.close();
+    }
+}));
    +
    +
    +
    +

    The last step is to execute the whole pipeline.

    +
    +
    +
    +
    +

    Running

    +
    +
    +

    We start by setting up the connector to relay the bucket contents into +the Kafka topic streaming-topic-beer-sample. It could be done either +using property files and connect-standalone as in +Quickstart, using REST interface of +connect-distributed or using Web UI provided by Control Center. We +will use the last two options.

    +
    +
    +

    By default Control Center starts at http://localhost:9021/. Connector +configuration is accessible in the "Kafka Connect" section:
    +image

    +
    +
    +

    Clicking on "New source" will open the configuration page of connectors. +Specify "Connection Name" as sample and "Connection Class" as +CouchbaseSourceConnector. Once the connector class is selected, the UI +will render a list of all accessible configuration properties:
    +image

    +
    +
    +

    The "Continue" button will lead to the next step where the form values +are converted into JSON, which can be used to define the connector using +the REST API:

    +
    +
    +
    +
    $ curl -X POST -H "Content-Type: application/json" http://localhost:8083/connectors \
+     --data '{
+               "name": "sample",
+               "connector.class": "com.couchbase.connect.kafka.CouchbaseSourceConnector",
+               "tasks.max": 2,
+               "connection.cluster_address": "localhost",
+               "connection.bucket": "beer-sample",
+               "topic.name": "streaming-topic-beer-sample"
+             }'
    +
    +
    +
    +

    Submitting the form (or using a REST call) will register the new +Connector link and start it immediately.

    +
    +
    +

    Now lets pull the sample sources and build them:

    +
    +
    +
    +
    $ git clone git://github.com/couchbase/kafka-connect-couchbase
+$ cd kafka-connect-couchbase
+$ mvn test-compile
+$ java -cp ./target/test-classes:$(mvn dependency:build-classpath | grep ^/) \
+       examples.KafkaStreamsDemo
    +
    +
    +
    +

    The records will start filling the beers and breweries tables.

    +
    +
    +

    Parent topic: Kafka Connector

    +
    +
    +

    Previous topic: +Sink Configuration Options

    +
    +
    +

    Next topic: Release Notes

    +
    +
    +
    +
    + + + +
    +
    +
    +
    +

    This page was built using the Antora default UI.

    +

    The source code for this UI is licensed under the terms of the MPL-2.0 license.

    +
    + + + + + diff --git a/content/connectors/spark-2.2/download-links.dita b/content/connectors/spark-2.2/download-links.dita index ea49961ef6..01c0bde2da 100644 --- a/content/connectors/spark-2.2/download-links.dita +++ b/content/connectors/spark-2.2/download-links.dita @@ -14,7 +14,7 @@ PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> Current Release 2.2.0

    The connector is currently compiled against both the Scala 2.10 and 2.11 targets to - comply with Spark. Here are the coordinates for both artifacts:

    + comply with Spark. Either of those artifacts can be used in Java applications. Here are the coordinates for both artifacts:

    • GroupId: com.couchbase.client
      • @@ -22,17 +22,17 @@ PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
    • Version: 2.2.0
    -

    If you are using Scala, here is the sbt snippet you can use:

    +

    If you are using Scala, here is the snippet you can use:

      -
    • 2.2.0: 2.2.0 (either of those JAR files can be used in Java applications): Download (Scala 2.10) | Download (Scala 2.10 / Java) | Download (Scala 2.11) | Download (Scala 2.11 / Java) | API Reference
    diff --git a/content/connectors/spark-2.2/getting-started.dita b/content/connectors/spark-2.2/getting-started.dita index 9766e32f55..eeafd652e3 100644 --- a/content/connectors/spark-2.2/getting-started.dita +++ b/content/connectors/spark-2.2/getting-started.dita @@ -64,13 +64,19 @@ val spark = SparkSession .builder() .appName("KeyValueExample") .master("local[*]") // use the JVM as the master, great for testing - .config("spark.couchbase.nodes", "127.0.0.1") // connect to couchbase on localhost - .config("spark.couchbase.bucket.travel-sample", "") // open the travel-sample bucket with empty password + .config("spark.couchbase.nodes", "127.0.0.1") // connect to Couchbase Server on localhost + .config("spark.couchbase.username", "Administrator") // with given credentials + .config("spark.couchbase.password", "password") + .config("spark.couchbase.bucket.travel-sample", "") // open the travel-sample bucket .getOrCreate() // The SparkContext for easy access val sc = spark.sparkContext]]> +

    The above example assumes Couchbase Server 5.X+ is being used, and hence a username and password + are required. If using an earlier version of Couchbase Server, the username and password lines + should be omitted and a non-empty bucket password may be required.

    +

    By default, the Spark Connector connects to the default bucket. However, this example uses actual data from the travel-sample bucket that ships with Couchbase Server, so the code connects to that bucket. The Couchbase Spark connector @@ -78,6 +84,53 @@ val sc = spark.sparkContext]]> +

    + Creating a JAR for use with spark-submit + +

    By compiling the application into a JAR file, it can be run with the standard Spark tool + spark-submit.

    + +

    In this case, the Spark dependencies can be declared as provided, as they will be + supplied by spark-submit:

    + + + +

    spark-submit allows the Spark session's configuration to be supplied on the command line, + so the Quickstart example above can be simplified to:

    + + + +

    The next step is to build a 'fat JAR' containing all the application's dependencies so it can easily be + run with spark-submit. One good option is to use the + sbt-assembly plugin.

    + +

    Finally, the created JAR can be run, supplying the required Couchbase configuration options:

    + + +
    +
    Creating and saving RDDs diff --git a/content/connectors/spark-2.2/working-with-rdds.dita b/content/connectors/spark-2.2/working-with-rdds.dita index 6c93c3c3bb..03dec9f48b 100644 --- a/content/connectors/spark-2.2/working-with-rdds.dita +++ b/content/connectors/spark-2.2/working-with-rdds.dita @@ -141,7 +141,7 @@ sc subset of the fields from a Document. This allows you to reduce network overhead, and you need to move less data between operations.

    -

    The couchbaseSubdocLookup method is available on both the SparContext and the RDD and accepts two/three arguments, depending on the context:

    +

    The couchbaseSubdocLookup method is available on both the SparkContext and the RDD and accepts two/three arguments, depending on the context:

    • ids: The list of IDs to fetch the fragments from.
      • diff --git a/content/data-access/data-access-intro.dita b/content/data-access/data-access-intro.dita index 79727993dc..da66e2da8b 100644 --- a/content/data-access/data-access-intro.dita +++ b/content/data-access/data-access-intro.dita @@ -56,7 +56,7 @@ can be used to handle queries about geometries, for example, to return a list of all items within a particular bounding box.

      MapReduce programs output either MapReduce Views or Spatial Views, which are described - further in .

      + further under Views.

    Querying with N1QL

    Couchbase Server can be programmed using SQL. Given that nearly all programmers already know @@ -121,6 +121,6 @@ continuously replicate data from the Couchbase Server cluster to the search engines.

    - + diff --git a/content/data-modeling/intro-data-modeling.dita b/content/data-modeling/intro-data-modeling.dita index 71f5be1bc8..2ee0c4cd56 100644 --- a/content/data-modeling/intro-data-modeling.dita +++ b/content/data-modeling/intro-data-modeling.dita @@ -14,17 +14,18 @@
  • Physical data modeling: In this phase you take the logical design and apply the entities and relationships to the containers provided to you by the Couchbase Server platform. Based on the access patterns, performance requirements, and atomicity and consistency requirements, you can choose which type of containers to use to represent your logical data model.

    • -->
    Relational Data Modeling vs Document Oriented Data Modeling -

    In a relational database system you must define a schema before adding records to a database. The schema is the structure described in a formal language supported by the database and provides a blueprint for the tables in a database and the relationships between tables of data. Within a table, you need to define constraints in terms of rows and named columns as well as the type of data that can be stored in each column.

    -

    In contrast, a document-oriented database contains documents, which are records that describe the "schema" of the data in the document, as well as the actual data. Documents can be as complex as you choose; you can use nested data to provide additional sub-categories of information about your object. You can also use one or more documents to represent a real-world object. The following example compares a conventional table with document-based objects.

    - - 1-to-many relationship comparison between relational table and JSON documents - - - -

    In this example we have a table that represents "hotels" and their respective attributes: name, address, city, and so forth. The relational model conforms to a schema with a specified number of fields which represent a specific purpose and data type. There is also a one-to-many relationship that is normalized to the "hotel reviews" table which also contains its own attributes from date of the review to the ratings. Normalization is common practice in the relational model that requires you to decompose data into smaller related tables

    -

    The equivalent document-based model has an individual document per "hotel". Each document contains the information for a specific "hotel" and embeds the "hotel reviews" in the document as an array.

    -

    In a document-oriented model, data objects are stored as documents; each document stores your data and enables you to update the data or delete it. Instead of columns with names and data types, you describe the data in the document, and provide the value for that description. If you want to add attributes to a "hotels" or "hotel reviews" table in a relational model, you need to modify the database schema to include the additional columns and their data types. In the case of document-based data, you just add the additional key-value pairs representing the new fields to your document.

    +

    In a relational database system you must define a schema before adding records to a database. The schema is the structure + described in a formal language supported by the database and provides a blueprint for the tables in a database and the + relationships between tables of data. Within a table, you need to define constraints in terms of rows and named + columns as well as the type of data that can be stored in each column. +

    + +

    In contrast, a document-oriented database contains documents, which are records + that describe the "schema" of the data in the document, as well as the actual data. + Documents can be as complex as you choose; you can use nested data to provide additional + sub-categories of information about your object. You can also use one or more documents to represent a real-world object. +

    +
    Using JSON Documents

    JavaScript Object Notation (JSON) is a lightweight data-interchange format which is easy to read and change. JSON is language-independent although it uses similar constructs to JavaScript. JSON documents enable you to benefit from all new @@ -56,7 +57,7 @@ "hotel_name":"Medway Youth Hostel", "reviews":[ { - "author":"author::Ozella_Sipes", + "author_id":"author::Ozella_Sipes", "content":"This was our 2nd trip here and we enjoyed it as much or more than last year. ", "date":"2013-06-22 18:33:50 +0300", "ratings":{ @@ -97,7 +98,7 @@ has the value "hotel".

  • What particular keys, ids, prefixes or conventions do you want to use for items, for instance ‘Hotel::Medway_Youth_Hostel.’
    • -
  • If want to use a document to access other documents. In other words, you can store keys that +
  • If you want to use a document to access other documents. In other words, you can store keys that refer other documents in a JSON document and get the keys through this document. For instance, the above example includes author_id under reviews that refer to the detailed information on authors.
    • diff --git a/content/data-modeling/modeling-relationships.dita b/content/data-modeling/modeling-relationships.dita index 3ac0566ae3..7c03a1190a 100644 --- a/content/data-modeling/modeling-relationships.dita +++ b/content/data-modeling/modeling-relationships.dita @@ -111,5 +111,23 @@

    The same pros and cons of embedding and referencing with 1-to-many relationships apply here. However, with many-to-many relationships, data duplication gets amplified by the cardinality of the relationships. This is true especially with embedding. Let's say due to your access pattern, you choose to embed "car" documents in a "driver" document. This would work well if there are only handful of "car" documents but they are very large with many attributes. On the other hand there are thousands of drivers per "car", you end up duplicating each "car" document thousands of time. In this case, it may make more sense to embed "drivers" in "car" document.

    Another way to model the relationship is to represent the relationship with a separate relationship document. This is similar to the relational modeling of the many-to-many relationships - these are called "mapping tables". For the "cars" and "drivers" example, this method produces a separate "car-driver" document that has a "car" document reference and a "driver" document reference for each car-driver combination. This typically isn’t a good approach in Couchbase Server as referencing and embedding provides a great deal of flexibility to avoid creating this redundant document.

    + +
    + + Using Sub-Document Operations + + +

    + The Couchbase SDK provides a Sub-Document API, which makes it unnecessary to transfer entire documents + over the network when only partial modifications are required. A sub-document is an inner component of + a JSON document. The sub-document is referred to by a path, which specifies attributes and array-positions. + Using paths, the Sub-Document API allows sub-documents to be read and written to. +

    + +

    + For more information, see + Sub-Document Operations. +

    +
    diff --git a/content/dev-guide.ditamap b/content/dev-guide.ditamap index 844395b1b3..4d2a069480 100644 --- a/content/dev-guide.ditamap +++ b/content/dev-guide.ditamap @@ -5,22 +5,22 @@ - + - + - + - + - + @@ -30,5 +30,7 @@ + + diff --git a/content/developer-guide/extended-attributes-fundamentals.dita b/content/developer-guide/extended-attributes-fundamentals.dita index 2ae58692cf..74eab42541 100644 --- a/content/developer-guide/extended-attributes-fundamentals.dita +++ b/content/developer-guide/extended-attributes-fundamentals.dita @@ -1,13 +1,13 @@ - Extended Attributes — Fundamentals + Extended Attributes: Fundamentals Couchbase Server permits the definition of extended attributes. These allow application developers to define application-specific metadata visible only to those applications that request it or attempt to modify it. - This might be, for example, meta-data specific to a programming framework that + This might be, for example, metadata specific to a programming framework that should be hidden by default from other frameworks, or possibly from other versions of the same framework. @@ -32,18 +32,19 @@ C. This is achieved by extensions to the - Subdocument API. + Subdocument API. As part of the Subdoc API, XATTR must necessarily be + JSON-specific, and are not able to store binary data directly.

    The creator-application can subsequently access and modify the extended attributes it has created within a document. - However, Couchbase Server 5.0 provides no facility for enumerating or sharing + However, Couchbase Server provides no facility for enumerating or sharing knowledge of extended attributes: therefore, no application has knowledge of the extended attributes within a document other than their creator; unless such knowledge is shared explicitly between applications by some mechanism external - to Couchbase Server 5.0. + to Couchbase Server.

    @@ -73,9 +74,9 @@

    A virtual extended attribute exposes additional information - about a document. Couchbase Server 5.0 provides a single virtual + about a document. Couchbase Server provides a single virtual extended attribute, $document, which, when specified - as a search-path, returns meta-data on the document. Ouput + as a search-path, returns metadata on the document. Output might appear as follows:

    diff --git a/content/developer-guide/query-consistency.dita b/content/developer-guide/query-consistency.dita index 409a287300..2039c6493d 100644 --- a/content/developer-guide/query-consistency.dita +++ b/content/developer-guide/query-consistency.dita @@ -37,6 +37,6 @@ query.consistency = CONSISTENCY_REQUESTExamples node.js

    - + diff --git a/content/developer-guide/sub-doc-api.dita b/content/developer-guide/sub-doc-api.dita index 02a87cf2aa..301ecf4199 100644 --- a/content/developer-guide/sub-doc-api.dita +++ b/content/developer-guide/sub-doc-api.dita @@ -74,9 +74,9 @@ LOGGER.info(resultLookup.content("name", String.class));

    same key. For example, the following operations do not collide although they are updating the same document.[Thread 1] - bucket.mutateIn("user").upsert("name.last","Lennon",false).doMutate(); + bucket.mutateIn("user").upsert("name.last","Lennon",false).execute(); [Thread 2] - bucket.mutateIn("user").upsert("email","jlennon@abc.com",false).doMutate();

    + bucket.mutateIn("user").upsert("email","jlennon@abc.com",false).execute();

    Commands

    This section lists the available sub-document commands. There are two categories of commands exposed through builder APIs in the SDKs:

    • lookupIn commands which are used to read data from existing documents.
      • diff --git a/content/eventing/eventing-Introduction.dita b/content/eventing/eventing-Introduction.dita new file mode 100644 index 0000000000..c9c572d680 --- /dev/null +++ b/content/eventing/eventing-Introduction.dita @@ -0,0 +1,333 @@ + + + + + Introduction + + Couchbase Eventing Service is a framework to operate on changes to data in real time. + Events are changes to data in the Couchbase cluster. + + +

      The term 'mutation' generally refers to document changes happening in the Couchbase cluster. + The Couchbase cluster considers create, update, expiry, or delete operations as a data + mutation. Natively integrated with the Couchbase Data Platform, the Eventing Service does not + require any third party solutions to manage data mutations. In an event-based architecture, + all data changes are reactive and in real-time.

      +

      For example, consider the number of mutations in the data cluster of an e-Commerce + application when a customer places a new order. You can observe that: a new transaction record + gets created, a new purchase-transaction gets appended to the User's account, sellers + stock-inventory gets updated with a new purchase order. Each of these data mutations qualifies + as an event in the cluster.

      +

      A few sample events are listed below:

      +

      +

        +
      • Capturing the sensor output from a patient
        • +
      • Placing an order in an e-Commerce application
        • +
      • A customer’s order causing a drop in the inventory
        • +
      • Change in the status of a project
        • +
      • Removing a user from a transaction scoring application
        • +
      • News article expires after a certain period of time
        • +
      +

      +

      Using the Eventing Service, you can:

      +

      +

        +
      • Set alerts in a document when a preconfigured threshold is breached
        • +
      • Monitor specific parameters in a document
        • +
      • Propagate changes to other systems
        • +
      • Enrich a document in real time
        • +
      • Cascade deletes to avoid orphaned documents
        • +
      +

      +
      Why use the Eventing Service? The Eventing + Service helps you to streamline your business workflows. Using the Couchbase Eventing Service + you can:
        +
      • Manage a data-driven business logic across business-critical applications in a timely + manner, thereby increasing your customer engagement.
        • +
      • Handle inconsistencies in business logic while working with multiple client applications + across domains.
        • +
      • Design newer data products by leveraging lower technical barriers in the Eventing + Service.
        • +
      • Develop a reliable infrastructure that can execute your business logic in a rapidly + changing data platform.
        • +
      • Scale your throughput without making changes to your data configuration and + infrastructure.
        • +
      • Maximize your return on investment by minimising your TCO.
        • +
      +
      + Comparing Eventing Service with Message Queue and Polling Implementations +

      The tabular columns list implementation of the Eventing Service in comparison with the + message queue and polling methods that are used to solve the tracking of data mutations.

      +
      + + Eventing Service and Message Queue or Bus Implementation + + + + + + Message Queue or Bus Implementation + Eventing Service Implementation + + + + + Needs an additional layer to propagate data changes. + Inherent to Couchbase Server and does not need a new layer to propagate data + changes. + + + Encounters dual-write problems. Every write operation gets pushed twice; once to + the message queue and the second time to the cluster. + Eliminates the dual-write problem. Multiple application servers can perform + simultaneous write operations. + + + At any point, a write-failure condition can happen. + Eliminates the write-failure condition; no transaction semantics. + + + No easy debug option during troubleshooting. + Integrated with native debugger support. + + + Leads to inefficient data governance and data leakages. + Provides a centralized control for aspects such as data auditing and data + governance, thereby reduces data leakages. + + + With the added license, infrastructure, and deployment expenses, the Total Cost + of Ownership (TCO) increases many folds. + No additional expenses, the TCO is reduced. + + + +
      +

      + + Eventing Service and Polling Implementation + + + + + + Polling Implementation + Eventing Service Implementation + + + + + To record and propagate data changes, multiple applications are needed. + Can record and propagate data changes to a database, message queue, an + end-point, or to another bucket inside the Couchbase cluster. + + + Batch systems are highly inefficient and are not reactive. + Handles data mutations in real-time. + + + Consumes a lot of CPU resources. + Implements as a state-less compute operation and utilizes latest trends in + compute (multi-core CPU). + + + Leads to code duplications across multiple infrastructure applications. + No code duplication. + + + Difficult to scale. You need to scale for applications and also scale for + transport layer requirements. + Provides easy horizontal scaling options. + + + +
      +

      +
      <b>Couchbase Functions</b>

      Couchbase + Functions, also referred to as Functions, offers a computing paradigm that developers can + use to handle data changes. In the Couchbase cluster, you can use the Functions to handle + data-changes according to an Event-Condition-Action model.

      For Application developers, + the Functions offers a platform using which you can focus on building the business logic, than + over configuration or infrastructure. Functions handle the event that is generated when a + document is created, updated, or deleted. The serverless computing infrastructure gets closer + to the datastore as the Couchbase Functions integrates with the Couchbase Data + Platform.
      +
      + Features of Functions +

      You can configure Functions to generate threshold-based alerts. If your business + logic requires you to monitor specific parameters inside a document, then you can use + Functions to trigger an alert upon a threshold breach.

      +

      Apart from notifications and alerts, Functions provides an option to propagate + data changes. You can propagate data changes and move documents to a different bucket inside + your Couchbase cluster. Since Couchbase is a NoSQL document-oriented data platform, + Functions provides an easy option to perform cascade deletes.

      +
      +
      + Benefits of Functions +
      +

      +

        +
      • Improves customer experience and engagement
          +
        • Data enrichment: Before the introduction of the Eventing Service, data enrichment + was accomplished through batch jobs. These batch jobs were not in real-time and often + resulted in increasing the cost of infrastructure and management. Using the Eventing + Service, the data enrichment capability was achievable in real-time. Functions involve + moderate coding effort, time to market and restart capabilities can be achieved + easily.
          • +
        • Simple to use: Since Functions are developed within the Eventing Service framework, + tracking data changes in your cluster is manageable.
          • +
        • +
      • Faster innovation
          +
        • With a focus on business logic, development cycles are reduced. The Eventing Service + platform offers a developer-friendly environment which in turn aids the faster + creation of Minimum-Viable-Products(MVPs).
          • +
        • Using Functions, Application Developers can rapidly remodel their business workflows + and thereby stay in-sync with any business change conditions.
          • +
        • Functions offer a lower barrier to technology-adoption by emphasizing on business + operations.
          • +
        • +
      • Reducing infrastructure and operations-cost
          +
        • Since the implementation of the Eventing Service is intrinsic to the Couchbase + cluster, it offers a simple to deploy working model.
          • +
        • The Eventing Service provides optimum utilization of resources and controls + essential aspects such as data auditing, data governance, and node scaling.
          • +
        • +
      +

      +
      + <b>Use Cases</b> +

      As an Organization, you can use the Eventing Service in a wide variety of use + cases. Be it in domains such as Retail, Healthcare, Telecom, Media, and Travel; you can + leverage the Eventing Service to track data mutations.

      +

      For an easy understanding, consider a sample use case in the Banking and + Financial domain. Let us say the user performs a credit card transaction. Using the + Event-Condition-Action model, you can design a custom workflow based on factors such as + user's credit limit, usage currency, and risk propositions.

      +

      As another sample use case, consider an organization operating in the Supply + Chain Management domain. As a developer, using the Function's Event-Condition-Action model, + you can design a custom workflow in your inventory for stock replenishment. Functions help + you to construct a business workflow that automatically triggers new stock replacements and + maintains a set stock threshold.

      +

      The tabular column below lists popular scenarios where the Eventing Service + across domains can be used.

      +

      + + Eventing Service - Use Cases + + + + + + + + Domain + Eventing Trigger + Condition Check + Sample Workflow + + + + + Banking & Financial Services + Card transaction + Transaction threshold + Generate risk alerts and quarantine user upon threshold + breach. + + + +

      Inventory/

      +

      Warehousing

      +       + New sales voucher + Stock availability + Generate invoice for stock replenishment. +       + + New purchase order + Saved wishlist/cart + Notify price alerts for wishlist items. + + + +

      Airline

      +       + New booking + Booking history + Enroll for frequent flyer program and notify special + promotions +       + + Enquiry + User profile + Notify price drop alerts + + + Healthcare + New report + Check for vitals + Schedule an appointment. + + + +

      Sports/

      +

      Gaming

      +       + New user creation + User profile + Generate notification about leaderboard and other + statistics. +       + + Media/ Entertainment + Breaking news + Query archives + Enrich existing news with archival information. + + + +
      +

      +
      +
      + <b><b>Eventing Service - Onboarding Information</b></b> +

      In your organization, if you are using the Couchbase data platform, then the Eventing + Service is a good fit for managing data mutations. The Multi-Dimensional Scaling (MDS) + service enables workload isolation and independent scalability in a Couchbase cluster. Using + the MDS service, the Eventing Service is in line with your real-time data tracking + requirements and integrates seamlessly with your existing Couchbase nodes such as the Query, + GSI, and FTS nodes. Adding a new Eventing node is a simple and repeatable process.

      +

      + + Eventing Service Onboarding Information +

      + +

      +

      Following are a few aspects during the onboarding process:

        +
      • The Eventing Service is intrinsic to Couchbase Server; unlike Polling and Message + Queue based external systems, it eliminates the need for an additional layer without + involving multiple applications for tracking data mutations. All data mutations are + handled in real-time, and the Eventing Service offers a centralized control for data + governance.
        • +
      • When you transition to leverage the Eventing Service, application developers can use + Couchbase Functions to manage business workflow changes swiftly. Application developers + can program, test, debug and troubleshoot on a single Eventing Service platform, instead + of managing multiple application across different network layers.
        • +
      • After onboarding, you can manage and optimize the system throughput efficiently. If + your data resides in the Couchbase cluster, based on aspects such as data workload, data + mutation rate, and Function execution latency, you can either use the scale out option + by adding additional workers or use the elastic scaling option by adding another node.
        • +
      • The Eventing Service provides an export and import option for code portability. Using + this option, you can reuse the Function handler code to validate the execution logic in + different environments with workload variations.
        • +
      • The Eventing Service is highly performant during the recursive restartability + operations. You can undeploy a Function, pause for few cycles and then start the + Function handler-code. Deploying a Function after a time-lapse ensures that the Function + execution is tested for restartability.
        • +
      • Eventing leverages the latest trends in multi-core CPUs; therefore nodes selected for + Eventing should optimally have a higher number of cores than those for indexing.
        • +

      +
      + +       diff --git a/content/eventing/eventing-Terminologies.dita b/content/eventing/eventing-Terminologies.dita new file mode 100644 index 0000000000..7523b0a25c --- /dev/null +++ b/content/eventing/eventing-Terminologies.dita @@ -0,0 +1,128 @@ + + + + Terminology + While using Functions, the following terminologies are used: + +
      FunctionsFunctions offers a computing paradigm + using which developers can handle data changes. Functions are either OnUpdate or + OnDelete. Resources are managed at or above a Function level, and the containing + Function scopes the state of all handlers.
      +
      + Handler +

      A stateless, short-running code-fragment that reacts to a specified event. One or more + handlers together constitute a Function. A handler must execute from start to finish, before + a specified timeout-duration. Currently, OnUpdate and OnDelete are the two + supported handlers. Couchbase Functions support multiple OnUpdate/OnDelete + handlers during the Function definition process.

      + +

      In the Couchbase Server 5.5 release, the handler code size is limited to 128KB. +

      +       +
      +
      + Statelessness +

      The persistent state of a Function is captured by entities listed below, while any state + that appears on the execution stack is ephemeral.

      +

      +

        +
      • The metadata bucket (which will eventually be a system collection).
        • +
      • The document being observed and its Extended Attributes.
        • +
      +

      +
      +
      + Buckets +

      Every Function definition needs two distinct buckets: source bucket and metadata bucket.

      +

      The Functions use a source bucket to listen to data changes. The Function handler code + polls the source bucket for data mutations. Multiple Functions can use the same source + bucket.

      +

      Metadata bucket stores internal artifacts and checkpoint information. The metadata bucket + provides better restartability semantics when the eventing nodes is offline. Ensure that + your Function handler code does not write any data to this bucket.

      +

      The Functions can in turn trigger data mutations, to avoid a cyclic generation of events, + ensure that you carefully consider options when you select the source and destination + buckets. When you are using a series of handlers, ensure that:

      +

      +

        +
      • The source bucket can be either Couchbase or Ephemeral bucket. However, memcached + buckets are not supported.
        • +
      • Metadata bucket is used by the Eventing Service to store some critical checkpoint + information. Avoid writing to the metadata bucket or flushing it. It is recommended that + a separate bucket be kept destined as a metadata bucket.
        • +
      • A read operation from the source bucket is allowed but operations such as delete, set + and update are not supported.
        • +
      • The destination buckets to which event handlers perform a write operation, do not have + other event handlers configured to listen and track data mutations.
        • +
      +

      +
      +
      + Workers +

      Workers can be defined as an execution unit that is assigned a group of vBuckets. The + worker units are used mostly during Function execution.

      +
      +
      + Bindings +

      Bindings are constructs that allow the separating of environment-specific variables from + the handler source code such as bucket-names, external endpoint URLs, and credentials. + Bindings allow Functions to be developed without source-code changes while working on + different workflows. Using bindings, you can port source code from different environments + such as Test-to-Test, Test-to-Production, or Production-to-Production environments without + making code changes.

      +

      During Function definition when you are creating a bindings name-value pair, for a source + bucket, ensure that only read operation is performed in the Function handler code.

      +
      +
      + Function Settings +

      The log levels, script timeout, and workers allocation are additional options available + during the Function definition process.

      +

      +

        +
      • Log Levels: Use log levels to determine the granularity of log messages.
        • +
      • Script Timeout: Script Timeout provides a timeout option to terminate a + non-responsive Function.
        • +
      • Workers: Workers allocate the number of worker threads for a specific Function. +

        For a defined Function, Couchbase Server limits the maximum number of workers to 10. + This upper limit is configured for system optimization purposes. In the Function + definition, if the number of workers exceeds a set threshold, then the system + automatically generates a warning message. However, this warning message does not + prevent deployment of the Function.

        • +
      +

      +
      +
      + Deploy +

      The deploy operation activates a Function. Multiple Functions can be deployed in a cluster. + Couchbase server performs source validations before deploying a Function, and only valid + Functions get deployed. The deploy operation transpiles the code and creates the executable + v8 artifacts. The source code of a deployed Function cannot be edited. The Function must be + in a deployed status to process events. Once you deploy a Function, depending on the Feed + Boundary conditions, the Function execution starts processing the data mutations. Deployment + of a Function triggers actions such as creating of necessary metadata, initiating worker + processes, calculating initial partitions, and creating checkpoints of processed stream + data. To edit a deployed Function, you must first un-deploy the Function.

      +
      +
      + Feed Boundary +

      Feed Boundary is a time milestone used during a Function configuration. Using the Feed + Boundary option, you can either invoke a Function on all the data present in the cluster or + choose to invoke a Function during future instances of data mutation, post Function + deployment.

      +
      +
      + Undeploy +

      An undeploy operation causes the Function to stop processing events. All allocated + worker-processes get terminated, and any runtime resources acquired by the Function gets + released. Functions in an undeployed state allow editing of the Function handler code. After + a successful undeploy operation, system does not retain the previous state. During + redeployment, the Eventing service cannot resume from a prior state before undeployment. An + undeployed Function retains the memory of its prior deployment, if necessary. Newly created + handlers start in an undeployed state.

      +
      +
      Delete When a Function gets deleted, the source + code implementing the Function, all processing checkpoints, and other artifacts in the + metadata provider are purged. Before deleting, make sure you have undeployed the Function. +
      + +       diff --git a/content/eventing/eventing-adding-function.dita b/content/eventing/eventing-adding-function.dita new file mode 100644 index 0000000000..efd1145bdd --- /dev/null +++ b/content/eventing/eventing-adding-function.dita @@ -0,0 +1,108 @@ + + + + + Adding a Couchbase Function + + To add a new Function, proceed as follows: + + +
      +
        +
      1. From the Couchbase Web Console > Eventing page, click ADD FUNCTION.
        2. +
      2. In the ADD FUNCTION dialog, for individual Function elements provide the below + information:

        + + Add a Function Dialog + + + + + + Elements + Description + + + + + Source Bucket + +

        The name of a bucket currently defined on the cluster.

        +         +         + + Metadata Bucket + +

        The name of a bucket currently defined on the cluster. This + bucket stores artifacts and checkpoint information.

        +         +         + + Function Name + A name, for the Function you are creating. + + + Description + A description of the Function you are creating. This is + optional. + + + Settings + +

        The available settings are:

        +
          +
        • Log Level: The granularity at which messages are logged. The + options (available from the arrows control at the right of the field) are + Info, Error, Warning, Debug, and (the default). This logging attribute is + function-specific and is not applicable for the Eventing Service in + general.
          • +
        • Workers: The number of worker threads to be allocated to the + Function. The default is 3.
          • +
        • Script Timeout: The number of seconds that should elapse before + the script times out. The default is 60.
          • +
        +         +         + + Bindings + +

        One or more strings, each of which is used as a reference to an + existing bucket, from within the JavaScript code of your Function.

        +

        For more information on Bindings, refer to .

        +         +         + + +
        +

        3. +
      3. Click Next: Add Code. The Function name dialog appears. The Function name dialog + initially contains a placeholder code block. You will substitute your actual Function code + in this block.
        4. +
      4. To return to the Eventing screen, click Eventing.
        5. +
      5. Click on the Function name. Additional controls are now displayed: The controls are:

        +

          +
        • Delete: Deletes the Function from the system.
          • +
        • + Export: Exports the Function as a JSON document.
          • +
        • Deploy: Deploys the Function, making it active across the cluster.
          • +
        • Edit JavaScript: Allows edits to be made on the Function, in an edit + dialog.
          • +
        +

        6. +
      6. Click Deploy. This displays the Confirm Deploy Function dialog. The Feed + Boundary determines whether documents previously in existence needs to be included in the + Function's activities: the options are Everything and From now. The + Everything option invokes a Function on all data mutations. The From now + option invokes a Function during future instances of data mutation, post Function + deployment.
        7. +
      7. Click Deploy Function. This deploys the Function and returns you to the main + Eventing screen. The display indicates that the Function is now deployed and running. From + this point, your defined Function will run, first, on all existing documents; and + subsequently, whenever a creation or mutation occurs.
        8. +
      8. Click Undeploy to undeploy a Function.
        9. +
      +
      + +       diff --git a/content/eventing/eventing-api.dita b/content/eventing/eventing-api.dita new file mode 100644 index 0000000000..dbb8a32790 --- /dev/null +++ b/content/eventing/eventing-api.dita @@ -0,0 +1,112 @@ + + + + Functions REST API + The Functions REST API, available by default at port 8096, provides the methods available to work with Couchbase + Functions. + +
      +

      + + Functions API + + + + + + + HTTP Method + URI Path + Description + + + + + POST + /api/v1/functions/[function_name] + Create a single Function. The Function name in the body must match + that on the URL. Function definition includes current settings. + + + POST + + /api/v1/functions/ + Creates multiple Functions. Function names must be unique. When + multiple Functions have the same name, an error is reported. + + + GET + + /api/v1/functions + View a list of Functions. Provides a list of Functions available in + the cluster. The list includes both the deployed and the undeployed + Functions. + + + DELETE + /api/v1/functions/[function_name] + Deletes a specific Functions from the cluster. + + + DELETE + +

      + /api/v1/functions

      +       + Deletes multiple Functions from the cluster. +       + + POST + /api/v1/functions/[function_name]/settings + Edit Function settings. During an edit, settings provided are + merged. Unspecified attributes retain their prior values. + + + POST + /api/v1/config + +

      Modify global configuration. During an edit, settings provided are + merged. Unspecified attributes retain their prior values. The response indicates + whether the Eventing service must be restarted for the new changes to take + effect.

      +       +       + + POST + /api/v1/functions/[function_name]/settings + +

      Deploys a Function. A deploy CURL example is provided for + reference.

      +

      Sample API: + curl -XPOST -d '{"deployment_status":true,"processing_status":true}' +http://Administrator@192.168.1.5:8096/api/v1/functions/[sample_name]/settings +

      +       +       + + POST + /api/v1/functions/[function_name]/settings + +

      Undeploys a Function. An undeploy CURL example is provided for + reference.

      +

      Sample API: + curl -XPOST -d '{"deployment_status":false,"processing_status":false}' +http://Administrator@192.168.1.5:8096/api/v1/functions/[sample_name]/settings

      +       +       + + POST + /api/v1/functions/[function_name]/settings + Deploys a Function with the provided code. + + + +
      +

      +
      + +       diff --git a/content/eventing/eventing-debugging-and-diagnosability.dita b/content/eventing/eventing-debugging-and-diagnosability.dita new file mode 100644 index 0000000000..a9e04f5ab6 --- /dev/null +++ b/content/eventing/eventing-debugging-and-diagnosability.dita @@ -0,0 +1,162 @@ + + + + Debugging and Diagnosability + Debugging and diagnostics in the Eventing Service comprises of debugging functions, + logging functions, and log redaction. + +
      <b>Debugging Functions </b>Couchbase Server, + for its Eventing Service framework, includes an online real-time Javascript Debugger. Debug is + a special flag on a Function. The Debug option integrates seamlessly with the Google Chrome + Debugger engine for running the Javascript code.

      Debugging Workflow +

      +

        +
      • During a debug session, a single mutation received by the Function is considered and + sent to the Debugger. This technique ensures that processing of the other data mutations + in the cluster does not get blocked.
        • +
      • The Worker in the Debugger pauses to trap the next event-instance, opens an ephemeral + TCP port, and generates a Chrome dev-tools URL, with a session cookie that controls the + debug worker.
        • +
      • All other events are unaffected except the trapped event-instance.
        • +
      • Using the Debug option, you can place breakpoints in the code and run the Function + execution, one line at a time. The step-step execution helps while troubleshooting the + deployed Function.
        • +
      • If the debugged event-instance completes execution, no further event-instance gets + trapped for debugging.
        • +
      • Debugging is a convenience-feature intended to help during Function development: it is + not designed for production environments. Debug mode should not be used in Product + environments, as it affects the in-order processing of the document mutations. In a + production environment, debug sessions introduce timing issues. If a debug session gets + terminated during execution, then the mutation may be abruptly processed.
        • +
      +

      Debugging a Function Using the Debug Option

      +

        +
      1. From the Couchbase Web Console > Eventing page, click on the name of a + deployed Function. The deployed Function expands to provide additional options. Click + Edit JavaScript.

        2. +
      2. From the Edit Function page, click Debug. A debug session gets + activated. As a result, the next event-instance gets trapped and is forwarded to the + Debugger.

        In the below screen, you can notice the message: "Waiting for mutation." +

        3. +
      3. Upon a data mutation, the debug URL in the Debugging pop-up gets + populated.

        4. +
      4. Copy the URL from the Debugging pop-up and paste it into your Google Chrome browser. +

        5. +
      5. From your Google Chrome browser, you can add breakpoints and run step-step diagnosis + to debug and troubleshoot the deployed Function. From the Debugging pop-up, click + Stop Debugging to terminate a debug session.
        6. +
      +

      +
      + Logging Functions +

      The Eventing Service creates two different types of logs:

        +
      • System Logs
        • +
      • Application Logs
        • +

      System Logs

      For the Eventing Service, Couchbase Server creates a separate + system log file, termed as eventing.log. The system log file captures all the Eventing + Service related system errors depending on the level and severity of the reported problem. + For every node, a single system log file gets created.

      The eventing.log + contains redactable user data and the log is collected using the cbcollect_info tool. + For log rotation, refer to . +

      Application Logs

      Application logs allow you to identify and capture + various Function related activities and errors.

      All Function-related activities such + as editing the handler code, debugging, or modifying feed boundaries conditions, get + recorded in the Application logs. Couchbase Server creates an individual log file for every + Function in the cluster. By default, the maximum size of an Application log file is 40MB, + and the number of log files before rotation is 10. Unlike system logs, the Application logs + are user configurable. You can access an Application log file using the command line + interface. Couchbase Server creates different application log files depending on the level + and severity of the reported problem, as configured during Function definition.

      + The cbcollect_info tool does not collect the Application log files. +

      + + Application Logs Path in Platform + + + + + + Platform + Location + + + + + Linux + /opt/couchbase/var/lib/couchbase/data/@eventing + + + Windows + C:\Program Files\Couchbase\Server\var\lib\couchbase\data\@eventing + (Assumes default installation location) + + + Mac OS X + /Users/<user>/Library/Application + Support/Couchbase/var/lib/couchbase/data/@eventing + + + +
      +

      To configure an Application log, use the REST endpoint settings option. +

      Sample URL: + 192.168.1.5:8091/_p/event/api/v1/functions/<Function_name>/settings +

      Sample Payload: + { + "settings": + { + "app_log_max_files": 10, + "app_log_max_size": 10485760 + } + }

      The sample payload above illustrates that the system stores 10 application log + files and each file records about 10 MB of data.

      At some point in time, old + application log files that are no longer necessary need to be deleted to make way for new + log records. When an Application log file reaches the set limit, a new log file gets + created. All the recorded information from the active log file gets transferred to this + newly created file.

      For illustration, consider enrich_ip_nums as the name of the + Function. A corresponding Application log file, enrich_ip_nums.log, gets created in the + Couchbase cluster. Whenever the enrich_ip_nums.log reaches 10MB in size, assuming the + maximum size of an Application log file is 10MB and the number of log files before rotation is + 10, the system automatically generates the enrich_ip_nums.log.1 file, during its first + iteration. The file enrich_ip_nums.log transfers all the log information to this new + log file. For this illustration, since the number of log files is 10, the system stores 10 + such files, the currently active log file along with 9 truncated files, at any given + instance.
      +
      Log Redaction +

      You can use logs for multiple purposes ranging from security, monitoring, and diagnostics. + Suppression of sensitive data such as personally identifiable information (PII), hostnames, + internal asset information, credit card details, during the logging operation is termed as + log redaction. Organizations implement log redaction as part of their legal compliance and + security risk mitigations.

      +

      Couchbase Server provides a capability to redact sensitive user data from getting captured in the logs. All sensitive data are scrubbed and gets removed from the log files. Post redaction, log files can be shared for troubleshooting without disregarding any regulatory compliance.

      +

      For details, see . +

      +
      + +       diff --git a/content/eventing/eventing-examples.dita b/content/eventing/eventing-examples.dita new file mode 100644 index 0000000000..22a0a63e94 --- /dev/null +++ b/content/eventing/eventing-examples.dita @@ -0,0 +1,379 @@ + + + + + Examples: Using the Eventing Service + + This page contains examples of how to use the Eventing Service, using the Couchbase + Web Console. + + +
      Example 1

      Goal: A document contains + attributes whose form makes them difficult to search on. Therefore, on the document's + creation or modification, a new attribute should be created to accompany each original + attribute; this new attribute being instantiated with a value that directly corresponds to + that of its associated, original attribute; but takes a different form, thereby becoming + more supportive of search. Original attributes are subsequently retrievable based on + successful searches performed on new attributes.

      Implementation: Create a + JavaScript function that contains an OnUpdate handler. The handler listens for + data-changes within a specified, source bucket. When any document within the bucket is created + or mutated, the handler executes a user-defined routine. In this example, if the created or + mutated document contains two specifically named fields containing IP addresses (these + respectively corresponding to the beginning and end of an address-range), the handler-routine + converts each IP address to an integer. A new document is created in a specified, target + bucket: this new document is identical to the old, except that it has two additional fields, + which contain integers that correspond to the IP addresses. The original document, in the + source bucket, is not changed.

      Preparations

      +

      This example requires the creation of three buckets: metadata, target and source buckets.

      +

      Proceed as follows:

      +
        +
      1. Create target and metadata buckets. To create a bucket, refer to Create a Bucket. The + target bucket contains documents that will be created by the Function. Don’t add any + documents explicitly to this bucket.
        2. +
      2. Follow Step 1. and create a source bucket. In the Source bucket screen:

        +

          +
        1. Click Add Document.
          2. +
        2. In the Add Document window, specify the name + SampleDocument as the NewDocument ID
          3. +
        3. Click Save.
          4. +
        4. In the Edit Document dialog, paste the following within the edit panel.

          + + + + + + +

          {

          +

          "country": "AD",

          +

          "ip_end": "5.62.60.9",

          +

          "ip_start": "5.62.60.1"

          +

          }

          +           +           + + +
          +

          5. +
        5. Click Save. This step concludes all required preparations.
          6. +
        +

        3. +
      +

      Procedure

      Proceed as follows:

        +
      1. From the Couchbase Web Console > Eventing page, click ADD FUNCTION, + to add a new Function.
        2. +
      2. In the ADD FUNCTION dialog, for individual Function elements provide the below information:

        +

          +
        • For the Source Bucket drop-down, select the source option + that was created for this purpose.
          • +
        • For the Metadata Bucket drop-down, select the metadata option that + was created for this purpose.
          • +
        • Enter enrich_ip_nums as the name of the Function you are creating in the + FunctionName text-box.
          • +
        • Enter Enrich a document, converts IP Strings to Integers that are stored in new + attributes, in the Description text-box.
          • +
        • For the Settings option, use the default values.
          • +
        • For the Bindings option, specify target as the name of the bucket, and + specify tgt as the associated value.
          • +
        +

        3. +
      3. After providing all the required information in the ADD FUNCTION dialog, click + Next: Add Code. The enrich_ip_nums dialog appears. The + enrich_ip_nums dialog initially contains a placeholder code block. You will + substitute your actual enrich_ip_nums code in this block.

        4. +
      4. Copy the following Function, and paste it in the placeholder code block of the + enrich_ip_nums dialog: + function OnUpdate(doc, meta) { + log('document', doc); + doc["ip_num_start"] = get_numip_first_3_octets(doc["ip_start"]); + doc["ip_num_end"] = get_numip_first_3_octets(doc["ip_end"]); + tgt[meta.id]=doc; +} + +function get_numip_first_3_octets(ip) +{ + var return_val = 0; + if (ip) + { + var parts = ip.split('.'); + + //IP Number = A x (256*256*256) + B x (256*256) + C x 256 + D + return_val = (parts[0]*(256*256*256)) + (parts[1]*(256*256)) + (parts[2]*256) + parseInt(parts[3]); + return return_val; + } +}

        After + pasting, the screen appears as displayed below:

        +

        The OnUpdate routine specifies that when a change occurs to data within the + bucket, the routine get_numip_first_3_octets is run on each document that + contains ip_start and ip_end. A new document is created whose data and + metadata are based on those of the document on which get_numip_first_3_octets is + run; but with the addition of ip_num_start and ip_num_end data-fields, + which contain the numeric values returned by get_numip_first_3_octets. The + get_numip_first_3_octets routine splits the IP address, converts each fragment + to a numeral, and adds the numerals together, to form a single value; which it + returns.

        5. +
      5. Click Save.
        6. +
      6. To return to the Eventing screen, click Eventing and click on the newly created + Function name. The Function enrich_ip_nums is listed as a defined Function. +

        7. +
      7. Click Deploy.
        8. +
      8. From the Confirm Deploy Function dialog, click Deploy Function. From this + point, the defined Function is executed on all existing documents and on subsequent + mutations.
        9. +
      9. To check results of the deployed Function, click the Documents tab.
        10. +
      10. Select target bucket from the Bucket drop-down.As this shows, a version of + SampleDocument has been added to the target bucket. It contains all the + attributes of the original document, with the addition of ip_num_start and + ip_num_end; which contain the numeric values that correspond to ip_start + and ip_end, respectively. Additional documents added to the source bucket, + which share the ip_start and ip_end attributes, will be similarly handled by + the defined Function: creating such a document, and changing any attribute in such a + document both cause the Function's execution.
        11. +
      +
      + Example 2 +

      Goal: This example illustrates how to leverage Eventing Service in the + Banking and Financial domain. When a credit card transaction exceeds the user’s available + credit limit, to indicate a high-risk transaction, an alert can be generated.

      +

      This example requires you to create four buckets: flagged_transactions, + users, metadataand transactionsbuckets. For steps on + how to create buckets, see Create Bucket.

      +

      Implementation: Create a JavaScript Function that contains an + OnUpdate handler. The handler listens to data-changes within a specified, + transactions source bucket. When a document within the source bucket is + created or mutated, the handler executes a user-defined routine. In this example, if the + created or mutated document contains a high-risk transaction, a new document gets created in + a specified, flagged_transactions bucket.

      +

      Proceed as follows:

      +
        +
      1. From the Couchbase Web Console > Eventing page, click ADD + FUNCTION,to add a new Function. The ADD FUNCTION dialog appears.
        2. +
      2. In the ADD FUNCTION dialog, for individual Function elements provide the below + information:

        +

          +
        • For the Source Bucket drop-down, select transactions that was + created for this purpose.
          • +
        • For the Metadata Bucket drop-down, select metadata that was created for + this purpose.
          • +
        • Enter high_risks_transactions as the name of the Function you are + creating in the FunctionName text-box.
          • +
        • Enter Functions that computes risky transaction and flags them, in the + Description text-box.
          • +
        • For the Settings option, use the default values.
          • +
        • For the Bindings option, add two bindings. For the first binding specify + users as the name of the bucket, and specify user as the + associated value. For the second binding, specify + flagged_transactions as the name of the bucket, and specify + high_risk as the associated value.
          • +
        +

        3. +
      3. After providing all the required information in the ADD FUNCTION dialog, click + Next: Add Code. The high_risks_transactions dialog appears. +

        The high_risks_transactions dialog initially contains a placeholder code + block. You will substitute your actual high_risks_transactions code in this + block.

        4. +
      4. Copy the following Function, and paste it in the placeholder code block of the + high_risks_transactions + dialog:function OnUpdate(doc, meta) { + try + { + //log('txn id:', meta.id, '; user_id:', doc.user_id , ', doc.amount:', doc.amount); + var this_user = getUser(doc.user_id); + if (this_user) + { + if(this_user['creditlimit'] < doc.amount) + { + log('Txn['+String(meta.id)+']*****High Risk Transaction as Txn Amount:'+ String(doc.amount)+' exceeds Credit Limit:',this_user['creditlimit']); + doc["comments"] = "High Risk Transaction as Txn Amount exceeds Credit Limit " +String(this_user['creditlimit']); + doc["reason_code"] = "X-CREDIT"; + high_risk[meta.id] = doc; + return; + } + else + { + if(doc.txn_currency != this_user['currency']) + { + log('Txn['+ String(meta.id) +']*****High Risk Transaction - Currency Mismatch:'+ this_user['currency']); + doc["comments"] = "High Risk Transaction - Currency Mismatch:" + this_user['currency']; + doc["reason_code"] = "XE-MISMATCH"; + high_risk[meta.id] = doc; + return; + } + } + //log('Acceptable Transaction:',doc.amount, ' for Credit Limit:', this_user['creditlimit']); + } + else + { + log('Txn['+ String(meta.id) + "] User Does not Exist:" + String(doc.user_id) ); + } + } + catch (e) + { + log('Error OnUpdate :', String(meta.id), e); + } +} + +function OnDelete(meta) { + log('Document OnDelete:', meta.id); +} + +function getUser(userId) +{ + try + { + if(userId != null) + { + return user[userId]; + } + } + catch (e) + { + log('Error getUser :', userId,'; Exception:', e); + } + return null; +}

        After pasting, the screen appears as displayed below:

        The OnUpdate handler is triggered for every transaction. The + handler checks if the transaction amount is less than the user’s available credit limit. + When this condition is breached, then this transaction is flagged as a high-risk + transaction. The Function high_risks_transactions then moves this transaction to + a different bucket, flagged_transactions bucket. When the transaction is moved to + a new bucket, the handler enriches the document with predefined comments and also + provides a reason code. In the last part, the handler performs a currency + validation step. If the transaction currency is other than the preconfigured home + currency of the user, then the handler flags the transactions and moves it to a + different bucket.

        5. +
      5. Click Save.
        6. +
      6. To return to the Eventing screen, click Eventing.

        The Function high_risks_transactionsis + listed as a defined Function. Currently, it is listed as Undeployed and + Paused.

        7. +
      7. Click Deploy.
        8. +
      8. From the Confirm Deploy Function dialog, click Deploy Function. This + deploys the Function and displays the main Eventing screen. From this point, the + defined Function is executed on all existing documents and on subsequent mutations.
        9. +
      9. To check results of the deployed Function, after a sufficient time elapse, from the + Couchbase Web Console > Eventing page, click Buckets.
        10. +
      10. Click flagged_transactions bucket. All documents available in this bucket are + transactions that are flagged as high-risk transactions.

        This + indicates that transactions which were flagged as high risk gets moved to the + flagged_transactions bucket.

        11. +
      11. From the Couchbase Web Console > Query page, execute the below N1QL query: + SELECT reason_code, COUNT(1), num_txns, SUM(amount) amount +FROM `flagged_transactions` +GROUP BY reason_code;

        12. +
      +
      +
      + Example 3 +

      Goal: This example illustrates how to leverage the Eventing Service to + perform a cascade delete operation. When a user is deleted, Couchbase Functions provide a + reliable method to delete all the associated documents with the deleted user.

      +

      This example requires you to create three buckets: users, metadata and + transactions buckets.

      +

      For steps to create buckets, see Create Bucket.

      +

      Implementation: Create a JavaScript Function that contains an + OnDelete handler. The handler listens to data-changes within a specified, + users source bucket. When a user within the source bucket gets deleted, + the handler executes a routine to remove the deleted user. When the delete operation is + complete, all associated documents of the delete users get removed.

      +

      Proceed as follows:

      +
        +
      1. From the Couchbase Web Console > Eventing page, click ADD + FUNCTION,to add a new Function.

        2. +
      2. In the ADD FUNCTION dialog, for individual Function elements, provide the below + information:

        +

          +
        • For the Source Bucket drop-down, select the Users that was + created for this purpose.
          • +
        • For the Metadata Bucket drop-down, select the metadata that was + created for this purpose.
          • +
        • Enter delete_orphaned_txns as the name of the Function you are + creating in the FunctionName text-box.
          • +
        • Enter Delete Orphaned Transactions from the `transactions’ bucket when user_id + is less than 10 in the Description text-box.
          • +
        • For the Settings option, use the default values.
          • +
        • For the Bindings option, specify users as the name of the + bucket and specify src_user as the associated value.
          • +
        +

        3. +
      3. After providing all the required information in the ADD FUNCTION dialog, click + Next: Add Code. The delete_orphaned_txns dialog appears. +

        The delete_orphaned_txns dialog initially contains a placeholder code + block. You will substitute your actual delete_orphaned_txns code in this + block.

        4. +
      4. Copy the following Function, and paste it in the placeholder code block of the + delete_orphaned_txns screen: + function OnUpdate(doc, meta) { + log('OnUpdate document:', meta.id); +} + +function OnDelete(meta) { + log('Document Deleted:', meta.id); + if(meta.id < 10) + { + try + { + var this_user_id = meta.id; + var del = delete from `transactions` where user_id = TONUMBER($this_user_id); + del.execQuery(); + log('Deleted Orphaned Transactions for User:', this_user_id); + } + catch(e) + { + log('Exception:', e) + } + } +}

        After pasting, the screen appears as displayed below:

        The + OnDelete handler is triggered for user delete transaction. The handler checks if + the user_id is less than 10. When this condition is fulfilled, then an N1QL + query is triggered to delete all user related information. The handler is then configured + to record this delete operation in a Function specific application log file. .
        5. +
      5. To return to the Eventing screen, click Eventing. The Function + delete_orphaned_txns is listed as a defined Function. Currently, it is + listed as Undeployed and Paused.
        6. +
      6. Click Deploy.
        7. +
      7. From the Confirm Deploy Function dialog, click Deploy Function. From this + point, the defined Function is executed on all existing documents and on subsequent + mutations.
        8. +
      8. Navigate to the Couchbase Web Console > Query page. Before deleting a + user, a snapshot of Query Result from the users bucket is displayed: +

        9. +
      9. The Query Results display users with user_ids from 1 to 10.
        10. +
      10. Navigate to the Couchbase Web Console > Buckets page. Delete two users + from the Users bucket:

        +

          +
        • Select User4 from the list and click the delete icon.
          • +
        • Select User10 from the list and click the delete icon.
          • +
        +

        11. +
      11. From the Query Editor, execute an N1QL query to check that all related records + for the deleted users are removed from the cluster. + SELECT user_id, COUNT(1) FROM `Users` GROUP BY user_id ORDER BY user_id ASC;

        +
        12. +
      12. In the Query Results pane notice that user_ids, user_id4 and + user_id 10 are removed as part of the cascade user delete operation. +

        13. +
      +
      + + +       diff --git a/content/eventing/eventing-faq.dita b/content/eventing/eventing-faq.dita new file mode 100644 index 0000000000..5222439238 --- /dev/null +++ b/content/eventing/eventing-faq.dita @@ -0,0 +1,174 @@ + + + + + + Frequently Asked Questions + + + Some questions, frequently asked about the Eventing service and Functions. + + + +
      + + + Frequently Asked Questions + + +
        +
      • +

        What languages are supported?

        +

        Javascript is the language to be used while creating Functions. Node modules cannot be + imported: only simple Javascript can be used.

        +
        • +
      • +

        Is Eventing an MDS-enabled service?

        +

        Yes. MDS enables workload isolation in a Couchbase cluster. Eventing nodes can be added + as more Functions are added, or if the mutation rates are high.

        +
        • +
      • +

        Will all the updates to a document be seen in DCP?

        +

        When a document is updated multiple times in a small time-window, not all updates are + seen in DCP. This means that the event handlers see only those deduplicated events that + are published in DCP.

        +
        • +
      • +

        Is it supported for both text and binary formats of the documents?

        +

        The value or the document must always be text in the form of a JSON document.

        +
        • +
      • +

        Can we determine what has changed on the document?

        +

        No. But you can implement versioning as part of the application logic. If versioning is + important, include it as part of the data architecture.

        +
        • +
      • +

        What kind of node do I run my Eventing Service on?

        +

        Eventing leverages the latest trends in multi-core CPUs; therefore nodes selected for + Eventing should optimally have a higher number of cores than those for indexing.

        +
        • +
      • +

        How does the Functions offering compare with the Couchbase’s Kafka Connector?

        +

        The Functions offering is about server-side processing or compute; it does not require + any middleware to be deployed or managed. Couchbase’s Kafka connector is an SDK + component that needs an application container or middleware to run.

        +
        • +
      • +

        Are Functions similar to a Database Trigger?

        +

        In a rough sense, Functions are similar to the Post-Triggers of the database world. But + with Functions, the action is already completed at the data-layer, and the event handler + gives an interface by which developers can key in the logic of what needs to happen + ‘after’ the action is done. What a Function sees is the actual event of the change, and + hence it does not directly correlate with Database Triggers.

        +
        • +
      • +

        Are Functions similar to a Rules Engine?

        +

        Not exactly. A Rules Engine enforces ordering and other semantics that is not possible + out of the box with Functions. However, Functions can be used to implement one - even + Rules Engine. The Function in its purest form offers a rule to implemented closer to the + data, but cannot trigger another Function directly.

        +
        • +
      • +

        Are Functions similar to a Stored Procedure?

        +

        Stored Procedures enforce a request-response model. A stored procedure will not be + invoked automatically; it has to be invoked by a calling method. Functions are based on + the idea of events. Changes to the data are events. These events trigger Functions, and + hence this is not a request-response model in its purest sense.

        +
        • +
      • +

        Do I need a separate middleware to consume the Functions? How do I consume changes in + my middleware/application?

        +

        Database changes are consumed using the Function defined: there is no other + programmatic way of accessing the changes (such as by using an SDK, or some other form + of middleware). REST endpoints are exposed, to perform administrative operations.

        +
        • +
      • +

        Can I import my own JS libraries or Node Modules?

        +

        No. We do not allow import of node modules or external JS libraries.

        +
        • +
      • +

        Can a Function listen to all changes across Couchbase Server?

        +

        A defined Function listens only to changes that are exposed using the DCP for the + buckets (Couchbase and Ephemeral) in the data-nodes. Memcached buckets are not + supported. The Function cannot listen to changes happening in other Couchbase + components, such as FTS, Views, or GSI; nor can it listen to system events.

        +
        • +
      • +

        Can there be more than one Function listening to changes to a bucket?

        +

        Yes. More than one Function can be defined for the same bucket. This lets you process + the change according to the business logic that you enforce. But there is no ordering + enforced; for example, if bucket 'wine' has three different Functions, which are + FunctionA, FunctionB, and FunctionC, you cannot enforce the order in which these + Functions are executed. Also, database triggers suffered from scalability and + diagnosability issues. Functions offer multiple diagnosability solutions and is highly + scalable and performant.

        +
        • +
      • +

        What is the metadata bucket? Do I need to create a separate bucket?

        +

        To provide better restartability semantics when an Eventing node is offline, some + metadata needs to be stored: a Couchbase bucket solves this problem. Setting up the + metadata bucket is a one-time activity that is done cluster-wide. It is recommended that + the metadata bucket not be used for any other data-storage (which is to say, it should + not be accessed by any other application).

        +
        • +
      • +

        What is in the "meta" Function parameter (OnUpdate, OnDelete)? Is this the metadata we + currently write in order to figure out what has changed in the document?

        +

        These are the meta fields associated with the document. For more information, refer to + the Link section.

        +
        • +
      • +

        Are Functions supported for both KV and Document storage?

        +

        The Eventing Service listens to changes that appear in the DCP. DCP is valid for the + Data Service, and Functions operate on documents that are either in key-value or in + document (JSON) format.

        +
        • +
      • +

        Is it possible to get additional state during a Function execution? For example, can + you read from the data service in a Function to fetch related data? For example, can we + enrich the updated document with data from another document (using a document id)?

        +

        Yes. You can read from any other bucket, and enrich the document.

        +
        • +
      • +

        Can I get old and new values of the document inside the Function?

        +

        No. We do not support versioning of documents; therefore, this feature is not available + out of the box. Though customers can have another ‘Mother’ bucket that stores documents + that could be looked up, in order to determine the difference between the current + document and the last modified.

        +
        • +
      • +

        Does a rebalance have any effect on the firing of events?

        +

        No. Functions do not lose any mutations during a rebalance operations.

        +
        • +
      • +

        What happens to the Eventing Service during a failover condition?

        +

        When the Data Service experiences a failover condition, mutations may be lost and these + lost mutations are not processed by the Eventing service. When the Eventing node + experiences a failover condition, few mutations may be processed more than once.

        +
        • +
      • +

        Why can’t I create global variables?

        +

        We restrict the language model in such a way that chances of going wrong are minimized. + As Functions are stateless compute entities, global variables do not have a good + use-case, and therefore, they are not supported. Though you can define Javascript + functions inside a Function (outside the scope of OnUpdate and OnDelete) that can be + invoked any number of times from either of the event handlers.

        +
        • +
      • +

        Is the ordering of document mutations enforced?

        +

        All changes from a document are always processed in order.

        +
        • +
      • +

        What happens when a Function is debugged?

        +

        We block one of the mutations alone and hand it over to the debugger session. The rest + of the mutations continue to be serviced by the event handler.

        +
        • + +
      + +
      + + +       diff --git a/content/eventing/eventing-function-export.dita b/content/eventing/eventing-function-export.dita new file mode 100644 index 0000000000..81d18365a9 --- /dev/null +++ b/content/eventing/eventing-function-export.dita @@ -0,0 +1,55 @@ + + + + + Exporting Functions + + Couchbase provides an option to export Functions as a JSON document. Using the export + and import options, you can port defined Functions from your Test-to-Test, Test-to-Production, + or Production-to-Production environments. + + +

      To export a Function:

      +
        +
      1. From the Couchbase Web Console > Eventing page.
        2. +
      2. Select a Function to export. Click on the Function to view additional controls.
        3. +
      3. Click Export. The Function definition gets exported to your system.
        4. +
      +

      A sample exported file for the enrich_ip_nums function demonstrated in Example 1 is shown + below. Ensure that you refrain from editing an exported file before importing the file back to + the Couchbase cluster. + [ + { + "appcode": "function OnUpdate(doc, meta) {\n log('document', doc);\n doc[\"ip_num_start\"] = get_numip_first_3_octets(doc[\"ip_start\"]);\n doc[\"ip_num_end\"] = get_numip_first_3_octets(doc[\"ip_end\"]);\n tgt[meta.id]=doc;\n}\n\nfunction get_numip_first_3_octets(ip)\n{\n var return_val = 0;\n if (ip)\n {\n var parts = ip.split('.');\n\n //IP Number = A x (256*256*256) + B x (256*256) + C x 256 + D \nreturn_val = (parts[0]*(256*256*256)) + (parts[1]*(256*256)) + (parts[2]*256) + parseInt(parts[3]);\nreturn return_val;\n }\n}\n", + "depcfg": { + "buckets": [ + { + "alias": "tgt", + "bucket_name": "source" + } + ], + "metadata_bucket": "metadata", + "source_bucket": "source" + }, + "version": "evt-5.5.0-2938-ee", + "handleruuid": 1014383070, + "id": 0, + "appname": "enrich_ip_nums", + "settings": { + "cleanup_timers": false, + "dcp_stream_boundary": "everything", + "deadline_timeout": 3, + "deployment_status": true, + "description": "Enrich", + "execution_timeout": 1, + "log_level": "INFO", + "processing_status": true, + "user_prefix": "eventing", + "using_doc_timer": false, + "worker_count": 3 + }, + "using_doc_timer": false + } +]

      + +       diff --git a/content/eventing/eventing-language-constructs.dita b/content/eventing/eventing-language-constructs.dita new file mode 100644 index 0000000000..d4c5977495 --- /dev/null +++ b/content/eventing/eventing-language-constructs.dita @@ -0,0 +1,255 @@ + + + + Language Constructs + The language constructs are fundamental units of a language. This topic discusses the JavaScript constructs that have been removed and new constructs that have been added in order to support the requirements of Couchbase Functions. + + +

      Using JavaScript, you can write your custom Functions. Couchbase Functions inherit support for most ECMAScript constructs by using Google v8 as the execution container. However, to support the ability to shard and scale Function-execution automatically, some capabilities have been removed. + Additionally, to optimize language-utilization of the server environment, some new constructs have been added.

      + +
      + Removed Language Features

      The following JavaScript features have been + removed:

        +
      • Global State
        • +
      • Asynchrony
        • +
      • Browser and other Extensions
        • +

      + Global State:

      Functions do not allow global variables. All state must + be saved and retrieved from persistence providers. In Couchbase Server 5.5, the Data Service + provider is used as a persistence provider. Therefore, all global states are contained in + the Data Service bucket(s) made available to the Function through bindings. This restriction + is mandatory for the Function-logic to remain agnostic of the rebalance + operation.

      var count = 0; // Not allowed - global variable. +function OnUpdate(doc, meta) { + count++; +}

      Asynchrony: +

      Asynchrony, particularly asynchronous callback, to be Functional needs to retain + access to parent scope. This forms a node-specific, long-running state that prevents the + capture of entire long-running state in persistence providers. Therefore, Function handlers + are restricted to running as short-running, straight-line code, without sleep and wakeups. + Limited asynchrony is added back through time observers. Time observers are designed not to + make the state + node-specific

      function OnUpdate(doc, meta) { + setTimeout(function(){}, 300); // Not allowed - asynchronous flow. +}

      Browser + and other Extensions:

      .A Function executes on Couchbase Server similar to + the code that is visible in the browser.

      As a Developer, you need to understand that the + code displayed in the below example runs in the browser. The ‘window’ term in the code + window.XMLHttpRequest(), is not a server-side construct but is in the + context of a + browser.function OnUpdate(doc, meta) { + var rpc = window.XMLHttpRequest(); // Not allowed - browser extension. +}
      +
      + Added Language Features +

      The following constructs have been added:

        +
      • Bucket Accessors
        • +
      • Top level N1QL keywords such as SELECT, UPDATE, and INSERT
        • +

      + +

      Bucket Accessors:

      +

      Couchbase buckets, when bound to a Function, appear as a global JavaScript map. The map + operations such as get, set and delete are mapped to the Data Service providers get, + set, and delete operations respectively. Other advanced Data Service operations are + available as member Functions on the map object.

      + function OnUpdate(doc, meta) { + // Assuming 'dest' is a bucket alias binding + var val = dest[meta.id]; // this is a bucket GET operation. + dest[meta.id] = {"status":3}; // this is a bucket SET operation. + delete dest[meta.id]; // this is a bucket DEL operation. +} + +

      + N1QL:

      +

      Top level N1QL keywords, such as SELECT, UPDATE, and INSERT, are available as keywords + in Functions. Operations that return values are accessible through a special iterator on + which the for (var <row> of <iterator>) looping-construct has been defined. + This restricted looping-construct allows support of query-result streaming, and + automatic query-cancellation when the iterator goes out of scope. Any variable that is + reachable from the scope of the N1QL query can be referred to using + $<variable> syntax in the N1QL statement, where parameters get substituted + according to the rules of the named-parameter substitution in the N1QL grammar + specification.

      +

      + The N1QL queries in events are a BETA feature and may have some + rough edges and bugs, and may change significantly before the final GA release. This + Beta-release version of Couchbase Server 5.5 is intended for development purposes only; no + Enterprise Support is provided for Beta features. +

      +

      The iterator is an input iterator (elements are read-only). The keyword this + cannot be used in the body of the iterator. The variables created inside the iterator + are local to the iterator.

      + function OnUpdate(doc, meta) { + var strong = 70; + var stmt = + SELECT * // N1QL queries are embedded directly. + FROM `beer-samples` // Token escaping is standard N1QL style. + WHERE abv > $strong; // Local variable reference using $ syntax. + for (var beer of stmt) { // Stream results using 'for' iterator. + break; // Cancel streaming query by breaking out. + } +} + +
      +

      The Function handler code supports N1QL queries. Top level N1QL keywords, such as + SELECT, UPDATE, and INSERT, are available as keywords in Functions.

      +

      During deployment, if a handler code includes an N1QL query, then the system + generates a warning message. Warning Message: "Handler <function_name> uses + Beta features. Do not use in production environments."However, the warning + message does not prevent the Function deployment.

      +

      You must use $<variable>, as per N1QL specification, to use a + JavaScript variable in the query statement. The object expressions for substitution are not + supported and therefore you cannot use the meta.id expression in the + query statement.

      +

      Instead of meta.id expression, you can use var id = + meta.id in an N1QL query.

      +

      +

        +
      • Invalid N1QL query + DELETE FROM `transactions` WHERE username = $meta.id;
        • +
      • Valid N1QL query + var id = meta.id; +DELETE FROM `transactions` WHERE username = $id;
        • +
      +

      +

      When you use a N1QL query inside a Function handler, remember to use an escaped + identifier for bucket names with special characters (`bucket-name`). + Escaped identifiers are surrounded by backticks and support all identifiers in JSON

      +

      For example:

      +
        +
      • +

        If the bucket name is beer-sample, then use the N1QL query + such as: SELECT * FROM `beer-sample` WHERE type...

        +
        • +
      • +

        If bucket name is beersample, then use the N1QL query such + as: SELECT * FROM beersample WHERE type ...

        +
        • +
      +
      + Handler Signatures

      The Eventing Service supports two event-handlers:

        +
      • OnUpdate Handler
        • +
      • OnDelete Handler
        • +

      + OnUpdate Handler: This handler gets called when a document is created or + modified. The handler listens to data changes from the associated source vBucket. A sample + OnUpdate handler is displayed + below:

      function OnUpdate(doc, meta) { + if (doc.type === 'order' && doc.value > 5000) { + //‘phonverify’ is a bucket alias that is specified as a binding. + phoneverify[meta.id] = doc.customer; + } +}

      In + this handler following limitations exist:

        +
      • +

        If a document is modified several times in a short duration, the calls at times get + coalesced into a single event, due to deduplication.

        +
        • +
      • +

        It is not possible to distinguish a Create from an Update operation.

        +
        • +
      + OnDelete Handler:

      This handler gets called when a document is deleted. The + handler listens to data changes from the associated source vBucket. This handler also gets + invoked during the expiry of a document.

      A sample OnDelete handler is + displayed + below:

      function OnDelete(meta) { + var stmt = SELECT id from orders WHERE shipaddr = $meta.id; + for (var id of stmt) { + log("Address invalidated for pending order: " + id); + } + }

      In + this handler the following limitations exist:

        +
      • +

        It is not possible to distinguish a delete as a result of expiration from a + user-triggered delete operation.

        +
        • +
      • +

        It is not possible to get the value of the document that was just deleted or + the one that just got expired.

        +
        • +
      +
      + Reserved Words +

      Reserved words are words that cannot be used as a variable name, function name, + or as a property in the Function handler code. The following table lists the reserved words + that you must refrain from using as they are used by Couchbase's query language, N1QL.

      + + + + + + + + + + N1QL Keywords + + + + + ALTER + EXECUTE + MERGE + UPDATE + + + BUILD + EXPLAIN + PREPARE + UPSERT + + + CREATE + GRANT + RENAME + + + + DELETE + INFER + REVOKE + + + + DROP + INSERT + SELECT + + + + +
      + +

      What Happens If You Use a Reserved Word?

      +

      Let's say you try to create a new Function handler code using a reserved word for variable + names, for function names, and as a property bindings value. All three cases generate a + deployment error.

      +

      Reserved words as a variable name: + function get_numip_first_3_octets(ip) +{ + var grant = 0; + if (ip) + { + var parts = ip.split('.'); + } +}

      +

      Reserved words as a function name: + function grant(ip) +{ + var return_val = 0; + if (ip) + { + var parts = ip.split('.'); + } +}

      +

      During the Function deployment step, when the system validates the handler code, it + displays an error message such as the following: Sample Error Message - + Deployment failed: Syntax error (<line and column numbers>) - grant is a reserved name + in N1QLJs

      +

      Reserved words as a property bindings value

      +

      +
      + +       diff --git a/content/eventing/eventing-operations.dita b/content/eventing/eventing-operations.dita new file mode 100644 index 0000000000..16da4d5092 --- /dev/null +++ b/content/eventing/eventing-operations.dita @@ -0,0 +1,89 @@ + + + + + + Operations + + + + Eventing operations are exposed through the Couchbase Web Console, and by means of the + Couchbase CLI. + + + +
      + Deploy +

      This operation activates a function. Source validations are performed, and only valid + functions can be deployed. Deployment transpiles the code and creates the executable v8 + artifacts. The source code of an activated (deployed) function cannot be edited. Unless a + function is in deployed state, it will not receive or process any events. Deployment creates + necessary metadata, spawns worker processes, calculates initial partitions, and initiates + checkpointing of processed stream data.

      +
        +
      • + DCP Feed Boundaries: Deployment can be effected on a DCP provider in three + variations:
          +
        • + From Beginning: Currently affects only the DCP observer. Causes a DCP stream to + start from sequence number 0. Therefore, the DCP data observer causes the function to + visit each document at least once.
          • +
        • + From Now: Currently affects only the DCP observer. Causes functions to start + observing mutations from the current sequence number of the master of each vBucket. + Therefore, this causes functions to visit documents modified after it is deployed.
          • +
        • + From Prior Deployment: Currently affects only the DCP observer. This option + runs to the point where the prior function left off. In other words, documents that + are mutated during the time-period that starts when the prior version of the function + becomes undeployed, and ends when the new version becomes deployed, will be visited by + the new function.

          Note that this is not present in the Developer Preview build. +

          • +
        • +
      • + Timer Semantics: Timers are referred to as artifacts, as they reside outside the + scope of a function. Deployment can be affected by timers, which are provided in two + variants:
          +
        • + Retain Timers: Cause all queued timers from prior versions of the function to + fire on the latest version of the deployed function, when it is activated. Note that + the system will define an upper bound on the maximum number of timers that can be + accumulated in this manner.
          • +
        • + Cleanup Timers: Cause all timers created by prior function-versions to be + deleted: only timers created after deployment of the current function-version will be + honored.
          • +
        • +
      +
      +
      + Undeploy +

      This operation causes the function to stop processing events of all types, and shuts down + the worker-processes associated with the function. It releases any runtime resources + acquired by the function. Functions in undeployed state allow code to be edited. An + undeployed function retains memory of its prior deployment where necessary. Newly created + handlers start in undeployed state.

      +
      +
      + Delete +

      When a function is deleted, the source code implementing the function, all timers, all + processing checkpoints, and other artifacts in the metadata provider are purged. When + successively deployed functions are identically named, the subsequent, deployed function has + no relation to the prior, deleted function. Only undeployed functions can be deleted.

      +
      +
      + Debug +

      Debug is a special flag on a function: it causes the next event-instance received by the + function to be trapped and sent to a separate v8 worker with debugging enabled. The debug + worker pauses the trapped event-processing, opens an ephemeral TCP port, and generates a + Chrome devtools URL, with a session cookie that can be used to control the debug worker. All + other events except the trapped event-instance continue unencumbered. If the debugged + event-instance completes execution, no further event-instance is trapped for debugging until + the user elects to ‘Stop Debugging’; and opens a new debugging session, by clicking on the + ‘Debug’ button. Note that debugging is a convenience-feature intended to help during + function-development: it is not designed to be used on production environments. Moreover, + the Couchbase Server 5.5 integration of the v8 debugger is a Developer Preview feature only: + it does not provide correctness or functionality guarantees.

      +
      + +       diff --git a/content/eventing/eventing-overview.dita b/content/eventing/eventing-overview.dita new file mode 100644 index 0000000000..207cd0019b --- /dev/null +++ b/content/eventing/eventing-overview.dita @@ -0,0 +1,325 @@ + + + + + Eventing Service: Fundamentals + The Couchbase Eventing Service is a framework to operate on changes to data in real + time. Events are changes to data in the Couchbase cluster. + + +

      The term 'mutation' generally refers to document changes happening in the Couchbase cluster. + The Couchbase cluster considers create, update, expiry, or delete operations as a data + mutation. Natively integrated with the Couchbase Data Platform, the Eventing Service does not + require any third party solutions to manage data mutations. In an event-based architecture, + all data changes are reactive and in real-time.

      +

      For example, consider the number of mutations in the data cluster of an e-Commerce + application when a customer places a new order. You can observe that: a new transaction record + gets created, a new purchase-transaction gets appended to the User's account, sellers + stock-inventory gets updated with a new purchase order. Each of these data mutations qualifies + as an event in the cluster.

      +

      A few sample events are listed below:

      +

      +

        +
      • Capturing the sensor output from a patient
        • +
      • Placing an order in an e-Commerce application
        • +
      • A customer’s order causing a drop in the inventory
        • +
      • Change in the status of a project
        • +
      • Removing a user from a transaction scoring application
        • +
      • News article expires after a certain period of time
        • +
      +

      +

      Using the Eventing Service, you can:

      +

      +

        +
      • Set alerts in a document when a preconfigured threshold is breached
        • +
      • Monitor specific parameters in a document
        • +
      • Propagate changes to other systems
        • +
      • Enrich a document in real time
        • +
      • Cascade deletes to avoid orphaned documents
        • +
      +

      +

      Why use the Eventing Service?

      +

      The Eventing Service helps you to streamline your business workflows. Using the Couchbase Eventing Service + you can:

        +
      • Manage a data-driven business logic across business-critical applications in a timely + manner, thereby increasing your customer engagement.
        • +
      • Handle inconsistencies in business logic while working with multiple client applications + across domains.
        • +
      • Design newer data products by leveraging lower technical barriers in the Eventing + Service.
        • +
      • Develop a reliable infrastructure that can execute your business logic in a rapidly + changing data platform.
        • +
      • Scale your throughput without making changes to your data configuration and + infrastructure.
        • +
      • Maximize your return on investment by minimising your TCO.
        • +
      +

      Eventing Service versus Message Queue and Polling Implementations

      +

      The following table compares the implementation of the Eventing Service with the message + queue method that is used to solve the tracking of data mutations.

      + + + + + + + Message Queue or Bus Implementation + Eventing Service Implementation + + + + + Needs an additional layer to propagate data changes. + Inherent to Couchbase Server and does not need a new layer to propagate data + changes. + + + Encounters dual-write problems. Every write operation gets pushed twice; once to + the message queue and the second time to the cluster. + Eliminates the dual-write problem. Multiple application servers can perform + simultaneous write operations. + + + At any point, a write-failure condition can happen. + Eliminates the write-failure condition; no transaction semantics. + + + No easy debug option during troubleshooting. + Integrated with native debugger support. + + + Leads to inefficient data governance and data leakages. + Provides a centralized control for aspects such as data auditing and data + governance, thereby reduces data leakages. + + + With the added license, infrastructure, and deployment expenses, the Total Cost + of Ownership (TCO) increases many folds. + No additional expenses, the TCO is reduced. + + + +
      + +

      The following table compares the implementation of the Eventing Service with the polling + method that is used to solve the tracking of data mutations.

      + + + + + + + Polling Implementation + Eventing Service Implementation + + + + + To record and propagate data changes, multiple applications are needed. + Can record and propagate data changes to a database, message queue, an + end-point, or to another bucket inside the Couchbase cluster. + + + Batch systems are highly inefficient and are not reactive. + Handles data mutations in real-time. + + + Consumes a lot of CPU resources. + Implements as a state-less compute operation and utilizes latest trends in + compute (multi-core CPU). + + + Leads to code duplications across multiple infrastructure applications. + No code duplication. + + + Difficult to scale. You need to scale for applications and also scale for + transport layer requirements. + Provides easy horizontal scaling options. + + + +
      + +
      <b>Couchbase Functions</b>

      Couchbase + Functions, also referred to as Functions, offers a computing paradigm that developers can + use to handle data changes. In the Couchbase cluster, you can use the Functions to handle + data-changes according to an Event-Condition-Action model.

      For Application developers, + the Functions offers a platform using which you can focus on building the business logic, than + over configuration or infrastructure. Functions handle the event that is generated when a + document is created, updated, or deleted. The serverless computing infrastructure gets closer + to the datastore as the Couchbase Functions integrates with the Couchbase Data + Platform.
      + +

      Features of Functions

      +

      You can configure Functions to generate threshold-based alerts. If your business + logic requires you to monitor specific parameters inside a document, then you can use + Functions to trigger an alert upon a threshold breach.

      +

      Apart from notifications and alerts, Functions provides an option to propagate + data changes. You can propagate data changes and move documents to a different bucket inside + your Couchbase cluster. Since Couchbase is a NoSQL document-oriented data platform, + Functions provides an easy option to perform cascade deletes.

      + +

      Benefits of Functions

      + +
        +
      • Improves customer experience and engagement
          +
        • Data enrichment: Before the introduction of the Eventing Service, data enrichment + was accomplished through batch jobs. These batch jobs were not in real-time and often + resulted in increasing the cost of infrastructure and management. Using the Eventing + Service, the data enrichment capability was achievable in real-time. Functions involve + moderate coding effort, time to market and restart capabilities can be achieved + easily.
          • +
        • Simple to use: Since Functions are developed within the Eventing Service framework, + tracking data changes in your cluster is manageable.
          • +
        • +
      • Faster innovation
          +
        • With a focus on business logic, development cycles are reduced. The Eventing Service + platform offers a developer-friendly environment which in turn aids the faster + creation of Minimum-Viable-Products(MVPs).
          • +
        • Using Functions, Application Developers can rapidly remodel their business workflows + and thereby stay in-sync with any business change conditions.
          • +
        • Functions offer a lower barrier to technology-adoption by emphasizing on business + operations.
          • +
        • +
      • Reducing infrastructure and operations-cost
          +
        • Since the implementation of the Eventing Service is intrinsic to the Couchbase + cluster, it offers a simple to deploy working model.
          • +
        • The Eventing Service provides optimum utilization of resources and controls + essential aspects such as data auditing, data governance, and node scaling.
          • +
        • +
      +
      + <b>Use Cases</b> +

      As an Organization, you can use the Eventing Service in a wide variety of use cases. Be it + in domains such as retail, healthcare, telecommunications, media, or travel; you can + leverage the Eventing Service to track data mutations.

      +

      For an easy understanding, consider a sample use case in the banking and financial domain. + Let us say the user performs a credit card transaction. Using the Event-Condition-Action + model, you can design a custom workflow based on factors such as user's credit limit, usage + currency, and risk propositions.

      +

      As another sample use case, consider an organization operating in the Supply + Chain Management domain. As a developer, using the Function's Event-Condition-Action model, + you can design a custom workflow in your inventory for stock replenishment. Functions help + you to construct a business workflow that automatically triggers new stock replacements and + maintains a set stock threshold.

      +

      The tabular column below lists popular scenarios where the Eventing Service + across domains can be used.

      + + Eventing Service - Use Cases + + + + + + + + Domain + Eventing Trigger + Condition Check + Sample Workflow + + + + + Banking & Financial Services + Card transaction + Transaction threshold + Generate risk alerts and quarantine user upon threshold + breach. + + + +

      Inventory/

      +

      Warehousing

      +       + New sales voucher + Stock availability + Generate invoice for stock replenishment. +       + + New purchase order + Saved wishlist/cart + Notify price alerts for wishlist items. + + + +

      Airline

      +       + New booking + Booking history + Enroll for frequent flyer program and notify special + promotions +       + + Enquiry + User profile + Notify price drop alerts + + + Healthcare + New report + Check for vitals + Schedule an appointment. + + + +

      Sports/

      +

      Gaming

      +       + New user creation + User profile + Generate notification about leaderboard and other + statistics. +       + + Media/ Entertainment + Breaking news + Query archives + Enrich existing news with archival information. + + + +
      +
      +
      + Eventing Service - Onboarding Information +

      In your organization, if you are using the Couchbase data platform, then the Eventing + Service is a good fit for managing data mutations. The Multi-Dimensional Scaling (MDS) + service enables workload isolation and independent scalability in a Couchbase cluster. Using + the MDS service, the Eventing Service is in line with your real-time data tracking + requirements and integrates seamlessly with your existing Couchbase nodes such as the Query, + GSI, and FTS nodes. Adding a new Eventing node is a simple and repeatable process.

      +

      + + Eventing Service Onboarding Information +

      + +

      +

      Following are a few aspects during the onboarding process:

        +
      • The Eventing Service is intrinsic to Couchbase Server; unlike Polling and Message + Queue based external systems, it eliminates the need for an additional layer without + involving multiple applications for tracking data mutations. All data mutations are + handled in real-time, and the Eventing Service offers a centralized control for data + governance.
        • +
      • When you transition to leverage the Eventing Service, application developers can use + Couchbase Functions to manage business workflow changes swiftly. Application developers + can program, test, debug and troubleshoot on a single Eventing Service platform, instead + of managing multiple application across different network layers.
        • +
      • After onboarding, you can manage and optimize the system throughput efficiently. If + your data resides in the Couchbase cluster, based on aspects such as data workload, data + mutation rate, and Function execution latency, you can either use the scale out option + by adding additional workers or use the elastic scaling option by adding another node.
        • +
      • The Eventing Service provides an export and import option for code portability. Using + this option, you can reuse the Function handler code to validate the execution logic in + different environments with workload variations.
        • +
      • The Eventing Service is highly performant during the recursive restartability + operations. You can undeploy a Function, pause for few cycles and then start the + Function handler-code. Deploying a Function after a time-lapse ensures that the Function + execution is tested for restartability.
        • +
      • Eventing leverages the latest trends in multi-core CPUs; therefore nodes selected for + Eventing should optimally have a higher number of cores than those for indexing.
        • +

      +
      + +       diff --git a/content/eventing/eventing-statistics.dita b/content/eventing/eventing-statistics.dita new file mode 100644 index 0000000000..55a9b1d040 --- /dev/null +++ b/content/eventing/eventing-statistics.dita @@ -0,0 +1,41 @@ + + + + + + Statistics + + + Eventing Statistics, for each deployed Function, can be fetched from an Eventing node + using the Web Console. + + + +
      + +
      +

      Eventing Statistics, for each deployed Function, can be fetched from an Eventing + node using the Web Console.

      +

      +

        +
      1. To fetch Eventing Statistics for a deployed Function, from the Couchbase Web + Console > Servers page, click Statistics.
        2. +
      2. Click on a specific graph to expand. The Eventing Service Statistics provides four + options:
          +
        • Processed - displays the number of processed Functions.
          • +
        • Failures - displays the number of failures while processing the Function + handler code.
          • +
        • Backlogs - displays the number of mutations to be processed by a + Function.
          • +
        • Timeouts - displays the number of Functions which have encountered a timeout + condition.

          • +
        3. +
      +

      +
      +

      +
      + + +       diff --git a/content/eventing/eventing-terminology.dita b/content/eventing/eventing-terminology.dita new file mode 100644 index 0000000000..7498f02411 --- /dev/null +++ b/content/eventing/eventing-terminology.dita @@ -0,0 +1,137 @@ + + + + Terminology + While using Functions, the following terminologies are used: + +
      FunctionsFunctions offers a computing paradigm + using which developers can handle data changes. Functions are either OnUpdate or + OnDelete. Resources are managed at or above a Function level, and the containing + Function scopes the state of all handlers.
      +
      + Handler +

      A stateless, short-running code-fragment that reacts to a specified event. One or more + handlers together constitute a Function. A handler must execute from start to finish, before + a specified timeout-duration. Currently, OnUpdate and OnDelete are the two + supported handlers. Couchbase Functions support multiple OnUpdate/OnDelete + handlers during the Function definition process.

      + +

      In the Couchbase Server 5.5 release, the handler code size is limited to 128KB. +

      +       +
      +
      + Statelessness +

      The persistent state of a Function is captured by entities listed below, while any state + that appears on the execution stack is ephemeral.

      +

      +

        +
      • The metadata bucket (which will eventually be a system collection).
        • +
      • The document being observed and its Extended Attributes.
        • +
      +

      +
      +
      + Buckets +

      Every Function definition needs two distinct buckets: source bucket and metadata bucket.

      +

      The Functions use a source bucket to listen to data changes. The Function handler code + polls the source bucket for data mutations. Multiple Functions can use the same source + bucket.

      +

      Metadata bucket stores internal artifacts and checkpoint information. The metadata bucket + provides better restartability semantics when the eventing nodes is offline. Ensure that + your Function handler code does not write any data to this bucket.

      +

      The Functions can in turn trigger data mutations, to avoid a cyclic generation of events, + ensure that you carefully consider options when you select the source and destination + buckets. When you are using a series of handlers, ensure that:

      +

      +

        +
      • The source bucket can be either Couchbase or Ephemeral bucket. However, memcached + buckets are not supported.
        • +
      • Metadata bucket is used by the Eventing Service to store some critical checkpoint + information. Avoid writing to the metadata bucket or flushing it. It is recommended that + a separate bucket be kept destined as a metadata bucket.
        • +
      • A read operation from the source bucket is allowed but operations such as delete, set + and update are not supported.
        • +
      • The destination buckets to which event handlers perform a write operation, do not have + other event handlers configured to listen and track data mutations.
        • +
      +

      +
      +
      + Workers +

      Workers can be defined as an execution unit that is assigned a group of vBuckets. The + worker units are used mostly during Function execution.

      +
      +
      + Bindings +

      Bindings are constructs that allow the separating of environment-specific variables from + the handler source code such as bucket-names, external endpoint URLs, and credentials. + Bindings allow Functions to be developed without source-code changes while working on + different workflows. Using bindings, you can port source code from different environments + such as Test-to-Test, Test-to-Production, or Production-to-Production environments without + making code changes.

      +

      During Function definition when you are creating a bindings name-value pair, for a source + bucket, ensure that only read operation is performed in the Function handler code.

      +
      +
      + Function Settings +

      The log levels, script timeout, and workers allocation are additional options available + during the Function definition process.

      +

      +

        +
      • Log Levels: Use log levels to determine the granularity of log messages.
        • +
      • Script Timeout: Script Timeout provides a timeout option to terminate a + non-responsive Function.
        • +
      • Workers: Workers allocate the number of worker threads for a specific Function. +

        For a defined Function, Couchbase Server limits the maximum number of workers to 10. + This upper limit is configured for system optimization purposes. In the Function + definition, if the number of workers exceeds a set threshold, then the system + automatically generates a warning message. However, this warning message does not + prevent deployment of the Function.

        • +
      +

      +
      +
      + Deploy +

      The deploy operation activates a Function. Multiple Functions can be deployed in a cluster. + Couchbase server performs source validations before deploying a Function, and only valid + Functions get deployed. The deploy operation transpiles the code and creates the executable + v8 artifacts. The source code of a deployed Function cannot be edited. The Function must be + in a deployed status to process events. Once you deploy a Function, depending on the Feed + Boundary conditions, the Function execution starts processing the data mutations. Deployment + of a Function triggers actions such as creating of necessary metadata, initiating worker + processes, calculating initial partitions, and creating checkpoints of processed stream + data. To edit a deployed Function, you must first un-deploy the Function.

      +
      +
      + Feed Boundary +

      Feed Boundary is a time milestone used during a Function configuration. Using the Feed + Boundary option, you can either invoke a Function on all the data present in the cluster or + choose to invoke a Function during future instances of data mutation, post Function + deployment.

      +
      +
      + Undeploy +

      An undeploy operation causes the Function to stop processing events. All allocated + worker-processes get terminated, and any runtime resources acquired by the Function gets + released. Functions in an undeployed state allow editing of the Function handler code. After + a successful undeploy operation, system does not retain the previous state. During + redeployment, the Eventing service cannot resume from a prior state before undeployment. An + undeployed Function retains the memory of its prior deployment, if necessary. Newly created + handlers start in an undeployed state.

      +
      +
      Delete When a Function gets deleted, the source + code implementing the Function, all processing checkpoints, and other artifacts in the + metadata provider are purged. Before deleting, make sure you have undeployed the Function. +
      +
      + Timers +

      Timers provide execution of code at a preconfigured clock time or after a specified number + of seconds.

      +

      Using timers, you can write a simple JavaScript Function handler code to delay or trigger + the execution of a Function at specific wall-clock time events. Additionally, you can + archive expired documents. Timers allow archiving of expired documents at a preconfigured + clock time.

      +
      + +       diff --git a/content/eventing/eventing.ditamap b/content/eventing/eventing.ditamap new file mode 100644 index 0000000000..3f1dd95953 --- /dev/null +++ b/content/eventing/eventing.ditamap @@ -0,0 +1,17 @@ + + + + + Eventing + + + + + + + + + + + + diff --git a/content/eventing/images/N1QL Query.png b/content/eventing/images/N1QL Query.png new file mode 100644 index 0000000000..e28eeaf3fc Binary files /dev/null and b/content/eventing/images/N1QL Query.png differ diff --git a/content/eventing/images/addCodePage.png b/content/eventing/images/addCodePage.png new file mode 100644 index 0000000000..9fa454bfec Binary files /dev/null and b/content/eventing/images/addCodePage.png differ diff --git a/content/eventing/images/addFunctionButton.png b/content/eventing/images/addFunctionButton.png new file mode 100644 index 0000000000..546cd68c5b Binary files /dev/null and b/content/eventing/images/addFunctionButton.png differ diff --git a/content/eventing/images/addFunctionDialog.png b/content/eventing/images/addFunctionDialog.png new file mode 100644 index 0000000000..252caca94a Binary files /dev/null and b/content/eventing/images/addFunctionDialog.png differ diff --git a/content/eventing/images/addTimerBeforeExpiryCode.png b/content/eventing/images/addTimerBeforeExpiryCode.png new file mode 100644 index 0000000000..21d0e34d9a Binary files /dev/null and b/content/eventing/images/addTimerBeforeExpiryCode.png differ diff --git a/content/eventing/images/add_functions_code_exp2.png b/content/eventing/images/add_functions_code_exp2.png new file mode 100644 index 0000000000..e55b3a2ea9 Binary files /dev/null and b/content/eventing/images/add_functions_code_exp2.png differ diff --git a/content/eventing/images/addfunctions-code_exp3.png b/content/eventing/images/addfunctions-code_exp3.png new file mode 100644 index 0000000000..718e7ddc7d Binary files /dev/null and b/content/eventing/images/addfunctions-code_exp3.png differ diff --git a/content/eventing/images/addfunctions.png b/content/eventing/images/addfunctions.png new file mode 100644 index 0000000000..035a232b4c Binary files /dev/null and b/content/eventing/images/addfunctions.png differ diff --git a/content/eventing/images/addfunctions_ex1.png b/content/eventing/images/addfunctions_ex1.png new file mode 100644 index 0000000000..bb6348ab09 Binary files /dev/null and b/content/eventing/images/addfunctions_ex1.png differ diff --git a/content/eventing/images/buckets.png b/content/eventing/images/buckets.png new file mode 100644 index 0000000000..4b550e26a4 Binary files /dev/null and b/content/eventing/images/buckets.png differ diff --git a/content/eventing/images/casacade_del_withcode.png b/content/eventing/images/casacade_del_withcode.png new file mode 100644 index 0000000000..1034416456 Binary files /dev/null and b/content/eventing/images/casacade_del_withcode.png differ diff --git a/content/eventing/images/cascade_delete_buckets.png b/content/eventing/images/cascade_delete_buckets.png new file mode 100644 index 0000000000..ceb0783e5f Binary files /dev/null and b/content/eventing/images/cascade_delete_buckets.png differ diff --git a/content/eventing/images/cascasde_delete.png b/content/eventing/images/cascasde_delete.png new file mode 100644 index 0000000000..df4f4888a8 Binary files /dev/null and b/content/eventing/images/cascasde_delete.png differ diff --git a/content/eventing/images/confirmDeployFunctionDialog.png b/content/eventing/images/confirmDeployFunctionDialog.png new file mode 100644 index 0000000000..cf8b5d8b07 Binary files /dev/null and b/content/eventing/images/confirmDeployFunctionDialog.png differ diff --git a/content/eventing/images/confirmDeployFunctionDialogDeployed.png b/content/eventing/images/confirmDeployFunctionDialogDeployed.png new file mode 100644 index 0000000000..dfd9f95664 Binary files /dev/null and b/content/eventing/images/confirmDeployFunctionDialogDeployed.png differ diff --git a/content/eventing/images/debug_1.png b/content/eventing/images/debug_1.png new file mode 100644 index 0000000000..136d168787 Binary files /dev/null and b/content/eventing/images/debug_1.png differ diff --git a/content/eventing/images/debug_2.png b/content/eventing/images/debug_2.png new file mode 100644 index 0000000000..23545024bb Binary files /dev/null and b/content/eventing/images/debug_2.png differ diff --git a/content/eventing/images/debug_3.png b/content/eventing/images/debug_3.png new file mode 100644 index 0000000000..e9eabf9cea Binary files /dev/null and b/content/eventing/images/debug_3.png differ diff --git a/content/eventing/images/debug_4.png b/content/eventing/images/debug_4.png new file mode 100644 index 0000000000..66081316e1 Binary files /dev/null and b/content/eventing/images/debug_4.png differ diff --git a/content/eventing/images/deploy_enrich_ip_nums.png b/content/eventing/images/deploy_enrich_ip_nums.png new file mode 100644 index 0000000000..d49bd89f43 Binary files /dev/null and b/content/eventing/images/deploy_enrich_ip_nums.png differ diff --git a/content/eventing/images/enrich_ip_nums.png b/content/eventing/images/enrich_ip_nums.png new file mode 100644 index 0000000000..2837b20499 Binary files /dev/null and b/content/eventing/images/enrich_ip_nums.png differ diff --git a/content/eventing/images/eventing-service-onboarding-information.jpg b/content/eventing/images/eventing-service-onboarding-information.jpg new file mode 100644 index 0000000000..2dd40a13dc Binary files /dev/null and b/content/eventing/images/eventing-service-onboarding-information.jpg differ diff --git a/content/eventing/images/eventing-service-onboarding-information.png b/content/eventing/images/eventing-service-onboarding-information.png new file mode 100644 index 0000000000..dfc99d6cac Binary files /dev/null and b/content/eventing/images/eventing-service-onboarding-information.png differ diff --git a/content/eventing/images/eventingScreenInitial.png b/content/eventing/images/eventingScreenInitial.png new file mode 100644 index 0000000000..8af11cc341 Binary files /dev/null and b/content/eventing/images/eventingScreenInitial.png differ diff --git a/content/eventing/images/eventingTab.png b/content/eventing/images/eventingTab.png new file mode 100644 index 0000000000..233bee9ecd Binary files /dev/null and b/content/eventing/images/eventingTab.png differ diff --git a/content/eventing/images/ex2AddFunctionDialogComplete.png b/content/eventing/images/ex2AddFunctionDialogComplete.png new file mode 100644 index 0000000000..00faaee6d3 Binary files /dev/null and b/content/eventing/images/ex2AddFunctionDialogComplete.png differ diff --git a/content/eventing/images/ex2metaDataDisplay.png b/content/eventing/images/ex2metaDataDisplay.png new file mode 100644 index 0000000000..b702157370 Binary files /dev/null and b/content/eventing/images/ex2metaDataDisplay.png differ diff --git a/content/eventing/images/ex2sampleDocument2.png b/content/eventing/images/ex2sampleDocument2.png new file mode 100644 index 0000000000..51b19611a8 Binary files /dev/null and b/content/eventing/images/ex2sampleDocument2.png differ diff --git a/content/eventing/images/ex2sourceBucketFinal.png b/content/eventing/images/ex2sourceBucketFinal.png new file mode 100644 index 0000000000..9d0fbcea49 Binary files /dev/null and b/content/eventing/images/ex2sourceBucketFinal.png differ diff --git a/content/eventing/images/ex2sourceBucketInitial.png b/content/eventing/images/ex2sourceBucketInitial.png new file mode 100644 index 0000000000..67eeb20bb7 Binary files /dev/null and b/content/eventing/images/ex2sourceBucketInitial.png differ diff --git a/content/eventing/images/ex2sourceBucketSubsequent.png b/content/eventing/images/ex2sourceBucketSubsequent.png new file mode 100644 index 0000000000..828e3e4fb9 Binary files /dev/null and b/content/eventing/images/ex2sourceBucketSubsequent.png differ diff --git a/content/eventing/images/ex2targetBucketSubsequent.png b/content/eventing/images/ex2targetBucketSubsequent.png new file mode 100644 index 0000000000..c1554196e0 Binary files /dev/null and b/content/eventing/images/ex2targetBucketSubsequent.png differ diff --git a/content/eventing/images/exAddFunctionDialogComplete.png b/content/eventing/images/exAddFunctionDialogComplete.png new file mode 100644 index 0000000000..0860710aa5 Binary files /dev/null and b/content/eventing/images/exAddFunctionDialogComplete.png differ diff --git a/content/eventing/images/functionAddedInitial.png b/content/eventing/images/functionAddedInitial.png new file mode 100644 index 0000000000..4c93705f39 Binary files /dev/null and b/content/eventing/images/functionAddedInitial.png differ diff --git a/content/eventing/images/functionAddedWithControls.png b/content/eventing/images/functionAddedWithControls.png new file mode 100644 index 0000000000..4e142b8626 Binary files /dev/null and b/content/eventing/images/functionAddedWithControls.png differ diff --git a/content/eventing/images/functions_add.png b/content/eventing/images/functions_add.png new file mode 100644 index 0000000000..48e84557b3 Binary files /dev/null and b/content/eventing/images/functions_add.png differ diff --git a/content/eventing/images/functions_add_4exp3.png b/content/eventing/images/functions_add_4exp3.png new file mode 100644 index 0000000000..45d2884026 Binary files /dev/null and b/content/eventing/images/functions_add_4exp3.png differ diff --git a/content/eventing/images/high_risks_transactions_handler_code.png b/content/eventing/images/high_risks_transactions_handler_code.png new file mode 100644 index 0000000000..e83c683850 Binary files /dev/null and b/content/eventing/images/high_risks_transactions_handler_code.png differ diff --git a/content/eventing/images/high_risks_transactions_handler_deploy.png b/content/eventing/images/high_risks_transactions_handler_deploy.png new file mode 100644 index 0000000000..2bc1f3dee9 Binary files /dev/null and b/content/eventing/images/high_risks_transactions_handler_deploy.png differ diff --git a/content/eventing/images/nextAddCodeButton.png b/content/eventing/images/nextAddCodeButton.png new file mode 100644 index 0000000000..bfa9a9bb16 Binary files /dev/null and b/content/eventing/images/nextAddCodeButton.png differ diff --git a/content/eventing/images/ondelete-functions.png b/content/eventing/images/ondelete-functions.png new file mode 100644 index 0000000000..cfeaa487b0 Binary files /dev/null and b/content/eventing/images/ondelete-functions.png differ diff --git a/content/eventing/images/pastedFunction.png b/content/eventing/images/pastedFunction.png new file mode 100644 index 0000000000..eca20435c6 Binary files /dev/null and b/content/eventing/images/pastedFunction.png differ diff --git a/content/eventing/images/query-results-ondelete.png b/content/eventing/images/query-results-ondelete.png new file mode 100644 index 0000000000..53f11870e4 Binary files /dev/null and b/content/eventing/images/query-results-ondelete.png differ diff --git a/content/eventing/images/queryresults_ondelerte.png b/content/eventing/images/queryresults_ondelerte.png new file mode 100644 index 0000000000..b7265e9612 Binary files /dev/null and b/content/eventing/images/queryresults_ondelerte.png differ diff --git a/content/eventing/images/reserved-words.png b/content/eventing/images/reserved-words.png new file mode 100644 index 0000000000..42297ced82 Binary files /dev/null and b/content/eventing/images/reserved-words.png differ diff --git a/content/eventing/images/selectSourceBucket.png b/content/eventing/images/selectSourceBucket.png new file mode 100644 index 0000000000..87664d4d5a Binary files /dev/null and b/content/eventing/images/selectSourceBucket.png differ diff --git a/content/eventing/images/settingsSubPanel.png b/content/eventing/images/settingsSubPanel.png new file mode 100644 index 0000000000..9a1458b0b3 Binary files /dev/null and b/content/eventing/images/settingsSubPanel.png differ diff --git a/content/eventing/images/stats.png b/content/eventing/images/stats.png new file mode 100644 index 0000000000..9409c186f0 Binary files /dev/null and b/content/eventing/images/stats.png differ diff --git a/content/eventing/images/targetBucketAfterEvent.png b/content/eventing/images/targetBucketAfterEvent.png new file mode 100644 index 0000000000..8c05d4272b Binary files /dev/null and b/content/eventing/images/targetBucketAfterEvent.png differ diff --git a/content/eventing/troubleshooting-best-practices.dita b/content/eventing/troubleshooting-best-practices.dita new file mode 100644 index 0000000000..496ccb6730 --- /dev/null +++ b/content/eventing/troubleshooting-best-practices.dita @@ -0,0 +1,102 @@ + + + + + <b>Troubleshooting and Best Practices</b> + +
      + <b>What happens when more Workers are allocated for a Function?</b> +

      Couchbase Server for a specific Function limits the maximum number of workers to + 10. This upper limit is configured for system optimization purposes. In the Function + definition, if the number of workers exceeds the set threshold, then the system + automatically generates a warning message. However, the warning message does not prevent the + Function deployment.

      +

      Warning Message: "There are <number_of_workers> eventing workers + configured to run. System performance may be impacted."

      +
      +
      + <b>Can the Couchbase Server 5.5 GA process cURL commands?</b> +

      Support for cURL commands in Functions is for demo purposes. Ensure not to use + cURL commands in your production environment.

      +

      The system generates a warning message. However, the warning message does not + prevent the Function deployment.

      +

      Warning Message: "Handler <function_name> uses Developer Preview + features. Do not use in production environments."

      +

      + cURL commands are a Developer Preview feature intended for + development purposes only, do not use them in production; no Enterprise Support is + provided for Developer Preview features. +

      +
      +
      + <b>When should developers use the try-catch block in Function handlers?</b> +

      As a best practice, while programming the Function handler code, for basic error + handling and debugging operations, it is recommended that application developers use the + try-catch block.

      +

      Before deployment, Couchbase Server verifies the Function handler code. Only + valid Functions get deployed. Using the log option within a try-catch block, you can record + errors. These error logs get stored in the application log file. By default, JavaScript + runtime errors get stored in the system logs. Unlike system logs, troubleshooting and + debugging operations are easy when you use the try-catch block and application log options.

      +

      A sample try catch block is provided for + reference:function OnUpdate(doc, meta) +{ + log('document', doc); + try { + var time_rand = random_gen(); + dst_bucket[meta.id + time_rand] = doc; + } + catch(e) + { + log(e); + } +}

      +
      +
      + <b>What are bucket allocation considerations during a Function definition?</b> +

      Function handlers can trigger data mutations. To avoid a cyclic generation of + data changes, ensure that you carefully consider the below aspects when you select the + source and destination buckets:

      +
        +
      • +

        Avoid infinite recursions. If you are using a series of handlers, then ensure + that destination buckets to which event handlers perform a write operation, do not have + other Function handlers configured to listen and track data mutations.

        +
        • +
      • +

        Couchbase Server can flag simple infinite recursions. However, in a long + chain of the source and destination buckets with a series of handlers, a complex + infinite recursion condition may occur. As a developer, carefully consider these cases + while allocating source and destination buckets.

        +
        • +
      • +

        As a best practice, ensure that buckets to which Function handlers perform a + write operation do not have other handlers configured for tracking data mutations.

        +
        • +
      +
      +
      + <b>In the cluster, I notice a sharp increase in the Timeouts Statistics. What are my + next steps? </b> +

      When the Timeout Statistics shows a sharp increase, it may be due to two possible + scenarios:

      +
        +
      • +

        Increase in execution time: When the handler execution time increases, the + Function execution latency gets affected, and this in turn, leads to Function backlog + and failure conditions.

        +
        • +
      • +

        Script timeout value: When the script timeout attribute value is not + correctly configured, then you encounter timeout conditions frequently.

        +
        • +
      +

      As a workaround, it is recommended to increase the script timeout value. Ensure + that you configure the script timeout value after carefully evaluating the execution latency + of the Function.

      +

      As a best practice use a combination of try-catch block and the application log + options. This way you can monitor, debug and troubleshoot errors during the Function + execution.

      +
      + +       diff --git a/content/fts/fts-creating-indexes.dita b/content/fts/fts-creating-indexes.dita index 5ba49a0fe5..3df80cf188 100644 --- a/content/fts/fts-creating-indexes.dita +++ b/content/fts/fts-creating-indexes.dita @@ -1,375 +1,1562 @@ - Text Indexing + + + Creating Indexes + + + + Full Text Searches are supported by specially purposed indexes, which can be created either from the + Couchbase Web Console, or by means of the REST API. + + -

      In order to perform full text queries, you must first create a full text index. - Unlike N1QL or other SQL systems, you can’t run full text searches directly on data without - a full text index: full text search has nothing like a table scan that can inspect documents - outside the index.

      -

      The process of creating an index is called index mapping, in which you specify - how you want Couchbase Server to map the different fields in your JSON documents to elements - in the full text index. You typically want a different index mapping for each type of document - you want to search. For example, if you’re using the Couchbase - beer-sample bucket, you might choose to map documents of type - beer one way and documents of type brewery - another.

      -

      An index can contain information from one bucket only. Your user must have appropriate access - permissions for the bucket they want to search. You can enable searches across buckets by - using an index alias.

      -

      To create an index, go to the Couchbase Server web console > Search - tab in the sidebar.

      -

      Your URL might look something like this, if you replace localhost with - the name of your server: - http://localhost:8091/ui/index.html#/fts_list

      -

      From here, you can create, edit, clone, and delete index definitions. If you see a message - saying "Full Text not enabled on this node," access the web - console on another node where the full text service is enabled.

      -

      To create an index in the web console, using your web browser:

        -
      • Navigate to the Search tab in the sidebar.
        • -
      • Click on the Add Index button.
        • -

      -

      To edit an index definition in the web console, using your web browser:

        -
      • Navigate to the Search tab.
        • -
      • This page displays any full text indexes that you have defined.
        • -
      Each listed index, when selected, has buttons for the following operations:
        -
      • The index edit button allows you to update an index definition. - Updating the index definition causes it to be rebuilt.
        • -
      • The index clone button allows you to copy an index - definition.
        • -
      • The index delete button allows you to delete an index definition. - When you click on the index delete button, you will have a - chance to first confirm the deletion operation. Deleting an index is a - permanent operation.
        • -

      -
      Index Mapping -

      The simplest way to create an index is to use the default index mapping with no further - customization. This is what you did in the section. Default index mapping - refers to the index mapping that Couchbase Server uses for JSON documents that don’t match a - more specific document mapping based on document type. This example relies on dynamic - mapping, explained later, to decide how to index the individual fields in the documents. - Using the default index mapping with dynamic mapping is a good way to - ensure that full text search is working properly on a small set of data, but it’s not very - selective so it tends to write a lot of information into the index. For this reason, - default index mapping may be slow or result in high load if used on a production-sized - dataset.

        -
      1. Click the button Add Index.
        2. -
      2. Give your index a name like "beer-sample-idx" and select the - bucket you want to index from the drop-down list.
        3. -
      3. Starting in version 4.6, you can create custom index mappings by document type when the - type is specified in the document key. FTS uses the type identifier that you specify to - determine where to find the type of each document. The type mapping then uses the value of - type identifier for each document to determine which index mapping rules to apply. Select - one of the three options for type identifier: "JSON type field", "Doc ID up to separator", - and "Doc ID with regex".
          -
        • JSON type field: Specify the field in the JSON document whose value - determines the type of the document. Defaults to "type".
          • -
        • Doc ID up to separator: The type identifier is the prefix of the document - key, up to but not including the given character.
          • -
        • Doc ID with regex: For advanced users, you can specify a regular expression - that matches the type identifier.
          • -

        For example, the Doc ID up to the first underscore is considered the type identifier, so beer name, brewery name, and so on could be specified as type mappings.

        4. -
      4. If you click on "Type Mappings," you will only see one type - listed, which is "default."
        5. -
      5. Click the "Create Index" button. You will then see a screen that - shows you how many documents are in your index and the percentage complete.
        6. -
      6. Click on the row for that index and you can test searching the index you just created using the search text box.
        7. -
      7. Your search shows the list of document IDs that contain the best matches for whatever search you ran, in order of score, with the highest scoring document listed first.
        8. -
      + +
      + + + Indexes and Full Text Search + + +

      + Every Full Text Search is performed on + a user-created Full Text Index, which contains the targets + on which searches are to be performed: these targets are values derived from the textual and + other contents of documents within a specified bucket. +

      + +

      + Index-creation is configurable, and can be highly selective: + documents can be user-grouped into + different types (for example, airline documents + versus hotel documents), based on the document-IDs or the values + of a designated document-field; and + each document-type can be assigned its own + index-mapping. Each index-mapping in turn can be assigned its own analyzers, can + be applied to a specific subset of document-fields, and can be explicitly included in + or excluded from the index. +

      + +

      + Additionally, searches can be performed across multiple buckets, by means + of index aliases. +

      + +

      + This section provides detailed explanations of how indexes can be created by means of the + Couchbase Web Console. Additionally, it explains how index-creation can be achieved with + the Couchbase REST API: see + Index-Creation with the REST API, + below. +

      +
      -
      Custom Mapping -

      There are many additional controls that can be used to build indexes on your documents just the - way you want them. These can be found under Search > Add Index button or Edit - when you select an existing index row.

      -

      Document mapping works as follows:

        -
      1. Specify the type of the document you want to include in your index. You can include more than - one type of document in a single index but you can only index documents in a single bucket.
        2. -
      2. For each type of document, you specify how to index its fields and embedded structures.
        3. -
      3. The term field refers to a name value pair in JSON where the value - is simple (i.e. not an object).
        4. -
      4. Use a child mapping to index embedded objects, i.e. name-value pairs where the - value is of type object. You can add field mappings as in step 3 to describe how the - name-value pairs in the embedded object should be indexed.
        5. -
      5. To index arrays, use fields for arrays of simple values and use a child mapping for - arrays of objects. In other words, you design your mapping so that you essentially - ignore the array and FTS "just works" even though there are multiple values. For - example, to index the following document containing an array, add a field of type - text for pachyderms. Queries work just like - any other field. If included in _all, a query for - rhinoceros matches this document, or you can scope to the field - using the normal field scoping syntax: pachyderms:rhinoceros. - { - "pachyderms": ["hippopotamus", "rhinoceros", "elephant"] -}
        6. -

      -

      Type Mappings

      -

      Couchbase Server indexes JSON documents differently depending on the type of the JSON document. - For example, you can create a full text index that only includes documents of - type="brewery," and you can specify exactly how you want the documents mapped. To do this, - click Add Type Mapping and enter a type name that matches the type - attribute of the JSON document you want to index.

      There is also a special type mapping for the Default Type. The default type mapping is created for every index automatically and is used for any document whose type does not match another type mapping or that doesn’t have a type attribute. You must disable the default mapping if you only want documents of the types that you specify to be in the index. If default mapping is enabled, Couchbase Server will use it to include all the documents in your index that don’t match another type mapping, which may or may not be what you want.

      -

      Each type mapping in an index definition can be enabled or disabled. Disabling a type - mapping can be used to ignore documents of a certain type. For example, if you want to index - all documents in the beer-sample bucket except "breweries" you could simply create a - type mapping for breweries and then check disabled (assuming the default mapping is also - disabled).

      -

      You can also specify an analyzer to use for a type mapping. This defaults to inheriting the - Default Analyzer specified in "Advanced".

      -

      Field Mapping

      -

      For any type mapping, you can insert a child field to index the values in your JSON - document with more control about what appears in the index and how. The word "field" in - index mapping refers to a name-value pair in JSON whose value is a simple type: string, - number, true, false, or null. These child fields refer to name-value pairs that are directly - under an object. In the brewery sample below, name, - city, and description are all fields.

      -

      You can index the description field of every document in the - beer-sample bucket, you can create an index, hover over the default index, click on - the plus that appears, and select "Insert child field". Because both - beers and breweries have a description field, this default mapping will end up with every - document in it. - { - "name": "21st Amendment Brewery Cafe", - "city": "San Francisco", - ... - "description": "The 21st Amendment Brewery offers a variety of award winning house made brews and American grilled cuisine in a comfortable loft like setting. Join us before and after Giants baseball games in our outdoor beer garden. A great location for functions and parties in our semi-private Brewers Loft. See you soon at the 21A!", - "address": [ - "563 Second Street" - ], - "geo": { - "accuracy": "ROOFTOP", - "lat": 37.7825, - "lon": -122.393 - } -}

      -

      There are four values and four checkboxes you can specify when you insert a child field.

        -
      • field: The name of name-value pair in the JSON document.
        • -
      • type: Defaults to text, but other possible values are object, - number, datetime, and disabled.
        • -
      • searchable as: You can change the name that is written into the - index, so if a user limits their search to a specific field they would use this value - instead of the actual name of the field in the JSON. For example, if we mapped the field - description "searchable as" info, instead of - typing "description:semi-private", users would instead search for "info:semi-private".
        • -
      • analyzer: The analyzer to use for this specific field.
        • -
      In addition, there are four checkboxes:
        -
      • index: If unchecked, fields that match this will not be - indexed. If the store checkbox is checked, they will still be stored.
        • -
      • store: Normally, only the document IDs are written to the - index. If this is checked, the document contents are also written to the index. This - enables highlighting and result snippets but generally results in larger indexes that - are slower to build. Since gets and multi-gets are quite fast, usually users don’t need - to store the additional information in the index.
        • -
      • Include in _all field: If this is checked, the text in this - field will be searchable in query strings without prefixing the field name. If - unchecked, the query must include this prefix, for example, "description:modern."
        • -
      • include term vectors: Term vectors are the locations of terms - in a particular field. Some functionality, such as snippets, highlighting, and phrase - search, requires term vectors and can’t be used without them. Not storing term vectors - results in smaller indexes and faster index build times.
        • -
      In this example, you would create the mapping like this: - field: description - type: text - searchable as: description (automatically filled in) - analyzer: inherit - Check "store" so that all four checkboxes are checked. This makes it - easier to test and debug your new index because search results will include snippets with - the search terms highlighted. The downside of storing the information is the extra size and - time it takes to build, but these should be acceptable in this case.

      -

      Field is the name of the name-value pair in the JSON. In our example, the - description property is at the top level but if you need to map - name-value pairs that are embedded in complex structures, you need to use a child mapping instead.

      -

      Insert Child Mapping

      -

      Child Mappings are similar to field mappings, but instead of indexing simple values, they enable you to index embedded structures in a JSON document. Use a child mapping when the value of a name-value pair is an object.

      -

      For example, consider the brewery document in the beer sample bucket. This document - contains an embedded object called "geo" that has three fields: accuracy, lat, and lon. - "geo": { - "accuracy": "ROOFTOP", - "lat": 37.7825, - "lon": -122.393 -} - To create an index with an object mapping for the geo structure using the Web Console, do - the following:

        -
      1. Create a type mapping for breweries.
        2. -
      2. Insert a child mapping for the attribute "geo".
        3. -
      3. In the "geo structure", insert a child field for - "accuracy".
        4. -
      4. Insert child fields for "lat" and "lon" and - set them to type number.
        5. -
      5. In a search on your newly created index, you can search the accuracy field using a dot - syntax: "geo.accuracy:rooftop".
        6. -
      6. You can also do range searches on the geo fields: "geo.accuracy:rooftop - +geo.lat:>37 +geo.lon:>141".
        7. -
      The index definition you created will look like the example below. You can create the - same index definition with object mappings using the REST API: - { - "type": "fulltext-index", - "name": "beer-idx", - "uuid": "af4c073428d51ed5", - "sourceType": "couchbase", - "sourceName": "beer-sample", - "sourceUUID": "a6e631d6c59692d8a8e4280615afd727", - "planParams": { - "maxPartitionsPerPIndex": 32, - "numReplicas": 0, - "hierarchyRules": null, - "nodePlanParams": null, - "pindexWeights": null, - "planFrozen": false - }, - "params": { - "mapping": { - "analysis": { - "analyzers": {}, - "char_filters": {}, - "token_filters": {}, - "token_maps": {}, - "tokenizers": {} - }, - "byte_array_converter": "json", - "default_analyzer": "standard", - "default_datetime_parser": "dateTimeOptional", - "default_field": "_all", - "default_mapping": { - "display_order": "1", - "dynamic": true, - "enabled": false, - "fields": [], - "properties": {} - }, - "default_type": "_default", - "type_field": "type", - "types": { - "brewery": { - "display_order": "0", - "dynamic": false, - "enabled": true, - "fields": [], - "properties": { - "geo": { - "display_order": "0", - "dynamic": true, - "enabled": true, - "fields": [], - "properties": { - "accuracy": { - "dynamic": false, - "enabled": true, - "fields": [ - { - "analyzer": "", - "date_format": null, - "display_order": "2", - "include_in_all": false, - "include_term_vectors": true, - "index": true, - "name": "accuracy", - "store": true, - "type": "text" - } - ], - "properties": {} - }, - "lat": { - "dynamic": false, - "enabled": true, - "fields": [ - { - "analyzer": "", - "date_format": null, - "display_order": "1", - "include_in_all": true, - "include_term_vectors": true, - "index": true, - "name": "lat", - "store": true, - "type": "number" - } - ], - "properties": {} - }, - "lon": { - "dynamic": false, - "enabled": true, - "fields": [ - { - "analyzer": "", - "date_format": null, - "display_order": "0", - "include_in_all": true, - "include_term_vectors": true, - "index": true, - "name": "lon", - "store": true, - "type": "number" - } - ], - "properties": {} - } - } - } - } - } - } - }, - "store": { - "kvStoreName": "forestdb" - } - }, - "sourceParams": { - "authPassword": "", - "authSaslPassword": "", - "authSaslUser": "", - "authUser": "beer-sample", - "clusterManagerBackoffFactor": 0, - "clusterManagerSleepInitMS": 0, - "clusterManagerSleepMaxMS": 2000, - "dataManagerBackoffFactor": 0, - "dataManagerSleepInitMS": 0, - "dataManagerSleepMaxMS": 2000, - "feedBufferAckThreshold": 0, - "feedBufferSizeBytes": 0 + +
      + + + Creating a Full Text Index + + +

      + Index-creation can be performed by means of the Couchbase Web Console. The basic + procedure is outlined in + Searching from the UI. + The current page explains the more + advanced aspects of index-creation, and shows how they can be + managed from the console. +

      + +
      + +
      + + + Accessing and Managing Full Text Indexes + + +

      + Full Text Indexes are different from the Global indexes that are + accessed under the Indexes tab in the left-hand navigation panel of the Couchbase + Web Console. Full Text Indexes are accessed from the Search tab: left-click on + this to display the Full Text Search panel, which + contains a tabular presentation of currently existing indexes, with a row for each + index. (See + Searching from the UI + for a full illustration.) + To manage an index, left-click on its row. The row expands, as + follows: +

      + +

      + +

      + +

      + Three buttons are displayed: +

      + +
        + +
      • + + Delete causes the current index to be deleted. + +

        + +

        + +
        • + +
      • + + Clone brings up the Clone Index screen, which allows the a copy + of the current index to + be modified as appropriate and saved under a new name. + +

        + + +

        + +
        • + +
      • +

        + Edit brings up the Edit Index screen, which allows + the index to be modified. + Saving modifications causes the index to be rebuilt. +

        + +

        + Note that both the Edit Index and Clone Index + screens are in most respects the same as the Add Index screen, + which was itself described in + Searching from the UI. +

        + +
        • + +
      + +
      + +
      + + + Specifying Type Identifiers + + +

      + A type identifier allows the documents in a bucket to be identified by the index + according to their type. When the + Add Index, Edit Index, or Clone Index + screen is accessed, a Type Identifier panel is displayed: +

      + +

      + +

      + +

      + Three options are provided, each of which gives the index a particular way of determining the type + of each document in the bucket: +

      + +
        + +
      • + JSON type field: The name of a document-field. The value specified + for this field is used by the index to determine the type of the document. + The default value is type: meaning that the index searches for a field + in each document whose name is type. Each document that + contains a field with + that name is duly included in the index, with the + value of the field specifying the type of the document. Note that the value cannot be + an array or JSON object. + +

        + +

        + +
        • + + +
      • + Doc ID up to separator: The characters in the ID of each document, + up to but not including the separator. For example, if the document's + ID is hotel_10123, the value hotel is + determined by the index to be the type of the document. The value entered into + the field should be the separator-character used in the ID: for example, + _, if that character is the underscore. + +

        + +

        + +
        • + +
      • + Doc ID with regex: A regular expression that is applied by the + index to the ID of each document. The resulting value is determined to + be the type of the document. (This option may be used when the targeted document-subset + contains neither a suitable JSON type field nor an ID that follows + a naming convention suitable for Doc ID up to separator.) The value + entered into the field should be the regular expression to be used. + +

        + +

        + +
        • + +
      + +
      + +
      + + + Specifying Type Mappings + + +

      + Whereas a type identifer tells the index how to determine + the position in each document of the characters + that specify the document's type, a type mapping specifies the + characters themselves. Thus, if Doc ID up to separator is used + as a type identifier, and the underscore is specified as the separator-character, + a type mapping of hotel ensures that + hotel_10123, rather than airline_10, is indexed. +

      + +

      + When the + Add Index, Edit Index, or Clone Index + screen is accessed, the Type Mappings panel can be opened. The + default setting is displayed: +

      + +

      + +

      + +

      + Left-click on the + Add Type Mapping button. The display now + appears as follows: +

      + +

      + +

      + +

      + The display indicates that a single type mapping is currently defined, which is + default. This is a special type mapping + created by every index automatically: it is applied to each document + whose type either does not match a user-specified type mapping, + or has no recognized type attribute. Therefore, if the default mapping is + left enabled, all documents are included in the index, regardless of + whether the user actively specifies type mappings. To ensure that only + documents corresponding to the user's specified type mappings are included + in the index, the default type mapping must be disabled (see below for an + example). +

      + +

      + Each type mapping is listed as either dynamic, meaning + that all fields are considered available for indexing; + or only index specified fields, meaning that + only fields specified by the user are indexed. + Therefore, specifying the default index with dynamic mapping + creates a large index whose response times may be relatively slow; and is, + as such, an option potentially unsuitable for post-production deployments. +

      + +

      + For information on how values are data-typed when dynamic mapping is specified, + see the section below, + Document Fields and Data Types. +

      + +

      + To specify a type mapping, type an appropriate string (for example, hotel) + into the interactive text field. Note the only index specified fields + checkbox: if this is checked, only user-specified fields from the document are + included in the index. (For an example, see + Specifying Fields, below.) +

      + +

      + Optionally, an analyzer can be specified for the type mapping: for all queries + that do indeed support use of an analyzer, the specified analyzer will be applied, + rather than the default analyzer (which is itself specified in the Advanced pane, as described + below, in + Specifying Advanced Settings). A list of + available analyzers can be accessed and selected from, by means of the pull-down + menu to the right of the interactive text-field: +

      + +

      + +

      + +

      + The default value, inherit, means that the type mapping inherits the + default analyzer. + Note that custom analyzers can be created and stored for the index that is being defined, + by means of the Analyzers panel, described below in + Creating Analyzers. + On creation, all custom analyzers are available for association with a type mapping, and so + appear in the pull-down menu shown above. +

      + +

      + Additional information on analyzers can also be found on the page + Understanding Analyzers. +

      + +

      + Left-click on OK to save. +

      + +

      + The Type Mappings panel now appears as follows: +

      + +

      + +

      + +

      + Note that the checkbox to the left of each of the two specified type mappings, + hotel and default, is checked. Because + default is checked, all documents in the bucket (not + merely those that correspond to the hotel type mapping) will be + included in the index. To ensure that only hotel documents + are included, uncheck the checkbox for default. The panel + now appears as follows: +

      + +

      + +

      + +

      + Note also that should you wish to ensure that all documents in the bucket are included + in the index except those that correspond to the hotel type + mapping, uncheck the checkbox for hotel: +

      + +

      + +

      + +
      + +
      + + + Specifying Fields + + +

      + A Full Text Index can be defined not only to include (or exclude) + documents of a certain type, + but also to include (or exclude) specified fields within each of the typed + documents. +

      + +

      + To specify one or more fields, hover with the mouse cursor over a row in the + Type Mappings panel that contains an enabled type mapping. Buttons labeled + edit and + appear: +

      + +

      + +

      + +

      + Left-clicking on the edit button displays the following interface: +

      + +

      + +

      + +

      + This allows the mapping to be deleted or associated with a different analyzer. If the + only index specified fields checkbox is checked, only fields specified by the user + are included in the index. Left-clicking + on the + button displays a pop-up that features two options: +

      + +

      + +

      + +

      + These options are described in the following sections. +

      + +
      + +
      + + + Inserting a Child Field + + +

      + The option insert child field allows a field to be individually + included for (or excluded from) indexing, provided that it contains a single value or an array, rather + than a JSON object. Selecting this option displays the following: +

      + +

      + +

      + +

      + The interactive fields and checkboxes are: +

      + +
        +
      • + field: The name of any field within the document that contains + a single value or an array, rather than a JSON object. + +

        + +

        + +
        • + +
      • + type: The data-type of the value of the field. This can + be text, number, datetime, boolean, + disabled, or geopoint; and can be selected from the + field's pull-down menu, as follows: + +

        + +

        + +
        • + +
      • + searchable as: Typically identical to the field (and + dynamically supplied during text-input of the field-value). This can be + modified, to indicate an alternative field-name, whose associated value thereby becomes included + in the indexed content, rather than that associated with the field-name specified in + field. + +

        + +

        + +
        • + +
      • + analyzer: An analyzer optionally to be used for the + field. The list of available analyzers can be displayed by means of the + field's pull-down muenu, and so selected from. + +

        + +

        +
        • + +
      • + index: When checked, the field is indexed; when unchecked, + the field is not indexed. This may be used, therefore, to explicitly remove an already-defined + field from the index. + +

        + +

        + +
        • + +
      • + store: When checked, the field-content is included in the set of + values returned from a search; when unchecked, the field-content is not so included. Note that + inclusion of field-content specifically permits highlighting of results, so that matched + expressions can be easily seen; and generally assists in debugging procedures. However, it also + results in larger indexes and + longer processing-times. + +

        + +

        + +
        • + +
      • + include in _all field: When checked, the field + is included in the definition of _all, which is + the field specified by default in the Advanced + panel. When unchecked, the field is not so + included. Inclusion means that when query strings are used to + specify searches, the text in the current field is searchable without + the field-name requiring a prefix (thus, a search on + description:modern can be accomplished simply by specifying + modern). + +

        + +

        + +
        • + +
      • + include term vectors: When checked, term + vectors are included. When unchecked, term vectors are not + included. Term vectors are the locations of terms + in a particular field. Certain kinds of functionality (such as highlighting, and phrase + search) require term vectors. Inclusion of term vectors + results in larger indexes and correspondingly slower index build-times. + +

        + +

        + +
        • + +
      + +

      + Note that when the value of the specified field is an array, the + array-values are all indexed and searched individually: no special + configuration is required. +

      + +

      + The dialog, when completed, might look as follows: +

      + +

      + +

      + +

      + Left-click on OK. The field is saved, and its principal attributes + displayed on a new row: +

      + +

      + +

      + +

      + Note that when this row is hovered over with the mouse, an Edit button appears, + whereby updates to the definition can be made. +

      + +
      + +
      + + + Inserting a Child Mapping + + +

      + The option insert child mapping specifies a document-field + whose value is a JSON object. Selecting this option + displays the following: +

      + +

      + +

      + +

      + The following interactive field and checkbox are displayed: +

      + +
        +
      • + {}: The name of a field whose value is a JSON object. Note that + an analyzer can be specified for the field, by means of the pull-down menu. + +

        + +

        + +
        • + +
      • + only index specified fields: When checked, only fields explicitly + specified are added to the index. Note that the JSON object specified as the value for + {} has multiple fields of its own. Checking this + box ensures that all or a subset of these can be selected for indexing. + +

        + +

        + +
        • + +
      + +

      + When completed, this panel might look as follows (note that reviews is + a field within the hotel-type documents of the travel-sample bucket + whose value is a JSON object): +

      + +

      + +

      + +

      + Save by left-clicking OK. The field is now displayed as part of the + hotel type mapping. Note that by hovering over the reviews + row with the mouse, the Edit and + buttons + are revealed: the + is present because reviews is + an object that contains child-fields; which can now themselves be individually indexed. Left-click on + this, and a child-field, such as content, can be specified: +

      + +

      + +

      + +
      + +
      + + + Creating Analyzers + + +

      + Analyzers increase search-awareness by transforming input text into token-streams, which + permit the management of richer and more finely controlled forms of text-matching. An + analyzer consists of modules, each of + which performs a particular role in the transformation (for example, removing + undesirable characters; transforming standard words into stemmed or otherwise modified forms, + referred to as + tokens; and performing miscellaneous post-processing activities). + For more information on analyzers, see + Understanding Analyzers. +

      + +

      + A default selection of analyzers is made available from the pull-down menu provided by the + Type Mappings interface, discussed above. Additional analyzers can + be custom-created, by means of the Analyzers panel, which appears + as follows: +

      + +

      + +

      + +

      + To create a new analyzer, left-click on the + Add Analyzer button. The + Custom Analyzer dialog appears: +

      + +

      + +

      + +

      + The dialog contains four interactive panels. +

      + +
        +
      • + Name: A suitable, user-defined name for the analyzer. + +

        + +

        +
        • + +
      • + Character Filters: One or more available character filters. (These + strip out undesirable characters from input: for example, the html + character filter removes HTML tags, and indexes HTML text-content alone.) To select from the + list of available character filters, use the pull-down menu: + +

        + +

        + +

        + Following addition of one character filter, to add another, left-click + on the + Add button, to the right of the field. +

        + +

        + For an explanation of character filters, see the section in + Understanding Analyzers. +

        +
        • + +
      • + Tokenizer: One of the available tokenizers. (These + split input-strings into individual + tokens, which together are made into a token stream. Typically, + a token is established for each word.) The default value is + unicode. To select from a list of all tokenizers available, + use the pull-down menu: + +

        + +

        + +

        + For more information on tokenizers, see the section in + Understanding Analyzers. +

        + +
        • + +
      • + Token Filter: One or more of the available token filters. (When specified, these + are chained together, to perform additional post-processing on the token stream.) To select + from the list of available filters, use the pull-down menu: + +

        + +

        + +

        + Following addition of one token filter, to add another, left-click + on the + Add button, to the right of the field. +

        + +

        + For more information on token filters, see the section in + Understanding Analyzers. +

        +
        • + +
      + +

      + When these fields have been appropriately completed, save; by left-clicking + on the Save button. On the Edit Index screen, + the newly defined analyzer now appears in the Analyzers panel, with + available options displayed for further editing, and deleting. For + example: +

      + +

      + +

      + +
      + +
      + + + Adding Custom Filters + + +

      + Custom Filters can be added, by means of the Custom Filters + panel. When opened, this appears as follows: +

      + +

      + +

      + +

      + The following four options are provided: +

      + +
        + +
      • + character filter: Adds a new character filter to the list of those available. The new + filter becomes available for inclusion in custom-created analyzers. Left-clicking on the + + Add Character Filter button displays the Custom Character Filter + dialog: + +

        + +

        + +

        + The following interactive fields are provided: +

        + +
          +
        • + Name: A suitable, user-defined name for the new character filter. + +

          + +

          +
          • + +
        • + Type: The type of filtering to be performed. Available options can be accessed + from the pull-down menu, at the right of the field. (Currently, only regexp + is available.) + +

          + +

          + +
          • + +
        • + Regular Expression: The specific regular expression that the new character filter + is to apply. Character-strings that match the expression will be affected, others will not. + +

          + +

          + +
          • + +
        • + Replacement: The replacement text that will be substituted for each character-string match + returned by the regular expression. If no replacement text is specified, the matched character-string + will be omitted. + +

          + +

          +
          • + +
        + + +

        + The following, completed fields define a character filter for deleting leading whitespace: +

        + +

        + +

        + +

        + When saved, the new character filter is displayed on its own row, with + options for further editing, and deleting: +

        + +

        + +

        + +
        • + +
      • + + tokenizer: Adds a new tokenizer to the list of those available. The new + tokenizer becomes available for inclusion in custom-created analyzers. Left-clicking on the + + Add Tokenizer button displays the Custom Tokenizer + dialog: + +

        + +

        + +

        + The following interactive fields are provided: +

        + +
          +
        • + Name: A suitable, user-defined name for the new tokenizer. + +

          + +

          + +
          • + +
        • + Type: The process used in tokenizing. Available options can be accessed from the pull-down menu, at the right of the field. + (Currently, regexp and exception are available.) + +

          + +

          + +
          • + +
        • + Regular Expression: The specific regular expression used by the tokenizing process. + +
          • + +
        + +

        + The following, completed fields define a tokenizer that removes uppercase characters: +

        + +

        + +

        + +

        + When saved, the new tokenizer is displayed on its own row, with options for + further editing, and deleting: +

        + +

        + +

        + +
        • + +
      • + + token filter: Adds a new token filter to the list of those available. The new token filter becomes avalable for inclusion in custom-created + analyzers. Left-clicking on the + Add Token Filter displays the Custom Token Filter dialog: + +

        + +

        + +

        + The following interactive fields are provided: +

        + +
          +
        • + Name: A suitable, user-defined name for the new token filter. +
          • + +
        • + Type: The type of post-processing to be provided by the new token filter. The default is length, which + creates tokens whose minimum number of characters is specified by the integer provided in the Min field, and whose + maximum by the integer provided in the Max. Additional post-processing types can be selected from the pull-down menu + at the right of the field: + +

          + +

          + +

          + Note that type-selection determines which interactive fields appear in the Custom Token Filter dialog, following Name + and Type. The pull-down menu displays a list of available types. For descriptions, see the + section + Token Filters, + on the page + Understanding Analyzers. +

          + +
          • + +
        • + Min: The minimum length of the token, in characters. Note that this interactive field is displayed for the length + type, and may not appear, or be replaced, when other types are specified. The default value is 3. +
          • + +
        • + Max: The maximum length of the token, in characters. Note that this interactive field is displayed for the length + type, and may not appear, or be replaced, when other types are specified. The default value is 255. +
          • + +
        + +

        + The following, completed fields define a token filter that restricts token-length to a minimum of 3, and a maximum of 255 characters: +

        + +

        + +

        + +

        + When saved, the new token filter is displayed on its own row, with options for + further editing, and deleting: +

        + +

        + +

        + +
        • + +
      • + + wordlist: Adds a list of words to be removed from the current search. Left-clicking on the + + Add Word List button displays the Custom Word List dialog: + +

        + +

        + +

        + To create a custom word list, first, type a suitable name into the Name field. Then, add words + by typing each individually into the field that bears the placeholder text, word to be added. After + each word has been added, left-click on the + Add + button, at the lower-right. The word is added to the central Words panel. Continue adding as many words as are required. For + example: +

        + +

        + +

        + +

        + To remove a word, select the word within the Words panel, and left-click on the Remove button. To + save, left-click on Save. The new word list is displayed on its own row, with + options for further editing, + and deleting: +

        + +

        + +

        + +
        • + +
      + +
      + +
      + + + Date/Time Parsers + + +

      + Date/Time Parsers can be specified, to allow matches to be made across different + formats: +

      + +

      + +

      + +

      + When the + Add Date/Time Parser button is left-clicked on, the + Customer Date/Time Parser dialog appears: +

      + +

      + +

      + +

      + Enter a suitable name for the custom parser into the Name field. Then, successively add the + layouts for the parser in the interactive field below the Layouts field, left-clicking on + the + Add button after each one: this adds the layout to a list of layouts displayed in the + Layouts field. To remove any of these, select its name in the Layouts field, and + left-click on the Remove button. When the list is complete, left-click on the Save + button, to save. +

      + +

      + Documentation on using the Go Programming Lanaguage to specify layouts is provided on the page + Package time. In particular, + see the section + func Parse. +

      + +
      + +
      + + + Specifying Advanced Settings + + +

      + Advanced settings can be specified by means of the Advanced panel. When opened, this appears as follows: +

      + +

      + +

      + +

      + The following, interactive fields are displayed: +

      + +
        + +
      • + + Type Field: The default type for documents in this bucket. The default value of this field is type. + +

        + +

        + +
        • + +
      • + + Default Type: The default type for documents in this bucket. The default value for this field is _default. + +

        + +

        + +
        • + +
      • + Default Analyzer: The default analyzer to be used for this bucket. The default value is standard. A list of + available options can be displayed and selected from, by means of the pull-down menu at the right-hand side of the field. + +

        + +

        + +
        • + +
      • + Default Date/Time Parser: The default date/time parser to be used for this bucket. The default value is dateTimeOptional. + A list of + available options can be displayed and selected from, by means of the pull-down menu at the right-hand side of the field. + +

        + +

        + +
        • + +
      • + Default Field: The default field for this bucket. the default value is _all. + +

        + +

        + +
        • + +
      • + Store Dynamic Fields: When checked, ensures inclusion of field-content in returned results. When unchecked, no such + inclusion occurs. + +

        + +

        + +
        • + +
      • + Index Dynamic Fields: When checked, ensures dynamic fields are indexed. When unchecked, they are not indexed. + +

        + +

        +
        • + +
      + +
      + + + Index Replicas + + +

      + The Index Replicas interface allows up to three index replicas to be selected, from + a pull-down menu: +

      + +

      + +

      + +

      + Index Replicas support availability: if an Index Service-node is lost from the cluster, its + indexes may exist as replicas on another cluster-node that runs the Index Service. If an + active index is lost, a replica is promoted to active status, and use of the index is + uninterrupted. +

      + +

      + Each replica must exist on a node separate from its active index, and from any other + replica of that active index. Attempts to add more than the number of replicas permitted + by the current cluster-configuration is not permitted, and results in an error message: +

      + +

      + +

      + + +
      + + + +
      + + + Index Type + + +

      + The Index Type interface provides a pull-down menu, from which the + appropriate index type can be selected: +

      + +

      + +

      + +

      + Two options are available: Version 5.0 (Moss) is the standard form of index, + to be used in test, development, and production; + Version 6.0 Preview (Scorch) is a preview of technology currently available + for test and development only. The Version 6.0 index reduces the size of + the index-footprint on disk, and provides enhanced performance for indexing and mutation-handling. +

      + +

      + Note that the type of an index is saved in its JSON definition, which can + be previewed in the + Index Definition Preview + panel, at the right-hand side. + Version 5.0 contains the following value for the store + attribute: +

      + + "store": { + "kvStoreName": "mossStore" + }, + +

      + Version 6.0 contains a different value: +

      + + "store": { + "kvStoreName": "", + "indexType": "scorch" +}, + +
      + +
      + + Using Index Aliases + +

      + An index alias points to one or more Full Text Indexes, or to additional aliases: + its purpose is therefore somewhat comparable to that of a symbolic link in a filesystem. Queries + on an index alias + are performed on all ultimate targets, and merged results are provided. +

      + +

      + The use of index aliases permits + indirection in naming, whereby applications refer to an alias-name that never changes, + leaving administrators free periodically to change the identity of the real index pointed + to by the alias. This may be particularly useful when an index needs to be updated: to avoid down-time, + while the current index remains in service, a clone of + the current index can be created, modified, and tested. Then, when + the clone is ready, the existing alias can be retargeted, + so that the clone becomes the current + index; and the (now) previous index can be removed. +

      + +

      + To create an Index Alias, access the Full Text Search screen, by left-clicking on the Search + tab, in the navigation bar at the left of the console. The Full Text Aliases panel is displayed in the lower section + of the page: +

      + +

      + +

      + +

      + Left-clicking on the Add Alias button displays the Add Alias screen: +

      + +

      + +

      + +

      + The Name field allows an appropriate name for the alias to be entered. The Target Indexes pane displays the + defined indexes available to be included in the alias. To select indexes within this pane, left-click on each: when the + index-name is highlighted, the index has been selected. +

      + +

      + To create the alias, left-click on the Create Index Alias button. The saved index now appears on its + own row in the Full Text Aliases area of the Full Text Search screen: +

      + +

      + +

      + +

      + Note that when the Add Alias screen is again accessed, by left-clicking the + Add Alias button, the travelSampleIndexesAlias appears + in the Target Indexes panel, along with the two existing indexes. +

      + +
      + + + +
      + + + Using the Index Definition Preview + + +

      + The Index Definition Preview appears to the right-hand side of the Edit Index screen. Following + index-definition, the upper portion may appear as follows: +

      + +

      + +

      + +

      + The preview consists of the JSON document that describes the current index-configuration, + as created by means of the user interface. By left-clicking on the + tab copy to clipboard, the + definition can be saved. By means of the REST API, the saved index definition (potentially after + modification) can be re-used in creation of an additional index: see the section immediately below. +

      + +
      + +
      + + + Index-Creation with the REST API + + +

      + The REST API can be used to create indexes. Each call requires the following: +

      + +
        + +
      • + An appropriate username and password. + +

        + +

        +
        • + +
      • + Use of the verb PUT. + +

        + +

        +
        • + +
      • + An endpoint referring to the Full Text Search service, on port 8094; and including the appropriate + endpoint for index-creation as defined by the + Full Text Search REST API, + including the name of the new index. + +

        + +

        +
        • + +
      • + Headers to specify settings for cache-control (no-cache) and application-type + (application/json). + +

        + +

        +
        • + +
      • + A body containing the JSON document that defines the index to be created. This must include the name of the bucket on which + the index is to be created. + +

        + +

        + +
        • + +
      + +

      + The simplest way to create the appropriate JSON index-definition for the body is to create an index by means of the Couchbase Web Console, make a copy + of the JSON index-definition thereby produced (by accessing the + Using the Index Definition Preview, + explained above), modify the index-definition as appropriate, and finally, add the index-definition to the other, preceding elements required for + the call. + + Note, however, that this requires modification of the uuid field; since the re-specifying of an existing field-value + is interpreted as an attempted update, to an existing index. Therefore, if the uuid field for an + existing index appears in the Index Definition Preview as "uuid": "3402702ff3c862c0", it should be edited to appear + "uuid": "". A new ID will be allocated to the new index, and this ID will appear in the Index Definition Preview for the new + index. Note also that a similar condition applies to the sourceUUID field, which refers to the targeted bucket: if a new + index is being created for the same bucket that was referred to in the index-object copied from the UI, the field-value can remain the same. However, + if a different bucket is now to be targeted, the field should be edited to appear "sourceUUID": "" +

      + +

      + When specifying the endpoint for the index you are creating, make sure the path-element that concludes the endpoint is the + same as that specified in the name field (which is the first field in the object). +

      + +

      + The following curl example demonstrates the creation of an index named demoIndex, on the + price field of documents of type product, within the travel-sample bucket. It + assumes that Couchbase Server is running on localhost, and that the required username and password + are Administrator and password. +

      + + $ curl -u Administrator:password -XPUT \ + http://localhost:8094/api/index/demoIndex \ + -H 'cache-control: no-cache' \ + -H 'content-type: application/json' \ + -d '{ +"name": "demoIndex", +"type": "fulltext-index", +"params": { + "doc_config": { + "docid_prefix_delim": "", + "docid_regexp": null, + "mode": "type_field", + "type_field": "type" + }, + "mapping": { + "default_analyzer": "standard", + "default_datetime_parser": "dateTimeOptional", + "default_field": "_all", + "default_mapping": { + "dynamic": true, + "enabled": false + }, + "default_type": "_default", + "index_dynamic": true, + "store_dynamic": false, + "types": { + "product": { + "dynamic": true, + "enabled": true, + "properties": { + "price": { + "enabled": true, + "dynamic": false, + "fields": [ + { + "analyzer": "", + "include_in_all": true, + "include_term_vectors": true, + "index": true, + "name": "price", + "store": false, + "type": "number" + } + ] } - }

      -

      Advanced Index Settings

      -

      Type field

      -

      By default, Couchbase Server will look for an attribute called "type" in your document and - will use that for type mappings. You can change the name of the "type" field by clicking on - the "Advanced" options and changing the value of the "Type - Field".

      -

      Default Type

      -

      Documents that match the default type mapping rules are written to the index as being of - this type. The field defaults to "_default", which you can change to avoid - conflicts if "_default" is a valid type field value in your JSON - documents.

      -

      Default Analyzer

      -

      This analyzer is used by default when creating an index, if no other analyzer is specified. - When you define your index mapping, you can override this value in a number of places, for - example, when you create type mappings or field mappings.

      -

      Default Date/Time Parser

      -

      The name of a Date/Time parser that will be used to parse a date stored as a string. - Full text search and Bleve expect dates to be in the format specified by RFC-3339, which is a specific profile of ISO-8601 that is more restrictive. -

      -

      Default Field

      -

      The default field is searched when a user query does not limit the scope of a search to a - particular field. This is highly convenient because most of the time you want a user to be - able to search for a term wherever it shows up in the index. For example, if you create a - text index of breweries, as long as terms are included in the _all field, - you can query "La Jolla" and find it without specifying "name:la - jolla" or "city:la jolla". You would only need to change the - name of this value if your JSON documents already include a field called - "_all".

      + } + } + } + }, + "store": { + "kvStoreName": "mossStore" + } +}, +"sourceType": "couchbase", +"sourceName": "travel-sample", +"sourceUUID": "99e9829898a45ba35f1c9c85dfcdb42b", +"sourceParams": {}, +"planParams": { + "maxPartitionsPerPIndex": 171, + "numReplicas": 0 + }, +"uuid": "" +}' + +

      + If the call is successful, the following object is returned: +

      + + {"status":"ok"} + +

      + The newly created index can then be inspected in the Couchbase Web Console. +

      +
      -
      Index Aliases -

      An index alias is a special "virtual index" that points to other, real full text indexes.

      Similar to a symbolic link in a file system, an index alias allows a naming level of indirection, so that applications can refer to a stable name (the alias' name) while administrators can dynamically re-target or re-point the index alias to different, real indexes. This can be useful for applications that are in production. For example, say you build an index my-index. You then build my-index-alias and point it to my-index. Your production application can search my-index-alias until you need to adjust the index, perhaps to change the index mapping or the analyzer. Any such change will require the index to be rebuilt, which would mean the current index will be out of commission. Instead, you can clone the index and give it a new name, like my-better-index. Once this index has had the time to build, you can test it before putting it into production. If you decide it's ready, you then modify the definition of my-index-alias to remove my-index and instead point to my-better-index. By using an alias like this, the switch to my-better-index will be instantaneous and the index will already be fully ready to use.

      -

      Similar to an email list alias, an index alias in FTS can also "fan-out" and refer to multiple, real indexes. Aliases can also refer to other aliases. A query on an index alias will be scatter-gathered by FTS to all of the actual, real indexes and FTS will provide merged results from those scatter-gathered queries.

      + +
      + + + Document-Fields and Data-Types + + +

      + During index-creation, for each document-field for which the data-type has not been + explicitly specified (which is to say, text, number, datetime, boolean, + disabled, or geopoint), the field-value is examined, and the best-possible + determination made, as follows: +

      + + + + + + + + Type of JSON value + Indexed as... + + + + + Boolean + Boolean + + + Number + Number + + + String containing a date + Date + + + String (not containing a date) + String + + + +
      + +

      + Note that the indexer attempts to parse String date-values as dates, and indexes them as such if + the operation succeeds. Note, however, that on query-execution, Full Text Search expects + dates to be in the format specified by + RFC-3339, + which is a specific profile of ISO-8601. +

      + +

      + Note also that String values such as 7 or true are + not respectively indexed as numbers or Booleans: they remain as Strings. +

      + +

      + The number-type is modeled as a 64-bit floating-point value internally. +

      +
      + + diff --git a/content/fts/fts-demonstration-indexes.dita b/content/fts/fts-demonstration-indexes.dita new file mode 100644 index 0000000000..c3a4088eec --- /dev/null +++ b/content/fts/fts-demonstration-indexes.dita @@ -0,0 +1,290 @@ + + + + + + Demonstration Indexes + + + + Demonstration indexes are provided, to exemplify the running of Full Text Searches. + + + + +
      + + + Using Demonstration Indexes + + +

      + An extensive code-example, which uses the Couchbase Java SDK, is provided in + Searching with the SDK. + This relies on three index definitions, respectively named travel-sample-index-unstored, travel-sample-index-stored, + and travel-sample-index-hotel-definition. Therefore, for the code-example to be successfully run, the + three indexes must be established on Couchbase Server. +

      + +

      + Optionally, the indexes can each be set up by means of the Couchbase Web Console, based on the descriptions provided in + Searching with the SDK. Information on + this process is provided in + Creating Indexes. However, each index can also be established + by means of a single REST command, which specifies all required index-details as a JSON document. Information on doing so can be found + in the section + Index-Creation with the REST API, on the + Creating Indexes page. The three definitions are provided below. +

      + +

      + This page also provides a definition for the geoIndex definition, used in the section + Geospatial Queries. +

      + +
      + +
      + + + Index Definition: travel-sample-index-unstored + + + { + "name": "travel-sample-index-unstored", + "type": "fulltext-index", + "params": { + "doc_config": { + "mode": "type_field", + "type_field": "type" + }, + "mapping": { + "analysis": { + "analyzers": { + "singleAnalyzer": { + "tokenizer": "single", + "type": "custom" + } + } + }, + "default_analyzer": "standard", + "default_datetime_parser": "dateTimeOptional", + "default_field": "_all", + "default_mapping": { + "dynamic": true, + "enabled": true + }, + "default_type": "_default", + "index_dynamic": true, + "store_dynamic": false, + "type_field": "type" + }, + "store": { + "kvStoreName": "mossStore" + } + }, + "sourceType": "couchbase", + "sourceName": "travel-sample", + "sourceUUID": "", + "sourceParams": {}, + "planParams": { + "maxPartitionsPerPIndex": 171, + "numReplicas": 0 + }, + "uuid": "" +} + +
      + +
      + + + Index Definition: travel-sample-index-stored + + + { + "name": "travel-sample-index-stored", + "type": "fulltext-index", + "params": { + "doc_config": { + "mode": "type_field", + "type_field": "type" + }, + "mapping": { + "analysis": { + "analyzers": { + "letterAnalyzer": { + "tokenizer": "letter", + "type": "custom" + } + } + }, + "default_analyzer": "standard", + "default_datetime_parser": "dateTimeOptional", + "default_field": "_all", + "default_mapping": { + "dynamic": true, + "enabled": true + }, + "default_type": "_default", + "index_dynamic": true, + "store_dynamic": true, + "type_field": "type" + }, + "store": { + "kvStoreName": "mossStore" + } + }, + "sourceType": "couchbase", + "sourceName": "travel-sample", + "sourceUUID": "", + "sourceParams": {}, + "planParams": { + "maxPartitionsPerPIndex": 171, + "numReplicas": 0 + }, + "uuid": "" +} + +
      + +
      + + + Index Definition: travel-sample-index-hotel-description + + + { + "name": "travel-sample-index-hotel-description", + "type": "fulltext-index", + "params": { + "doc_config": { + "mode": "type_field", + "type_field": "type" + }, + "mapping": { + "analysis": { + "analyzers": { + "myUnicodeAnalyzer": { + "tokenizer": "unicode", + "type": "custom" + } + } + }, + "default_analyzer": "standard", + "default_datetime_parser": "dateTimeOptional", + "default_field": "_all", + "default_mapping": { + "dynamic": true, + "enabled": false + }, + "default_type": "_default", + "index_dynamic": true, + "store_dynamic": true, + "type_field": "type", + "types": { + "hotel": { + "dynamic": false, + "enabled": true, + "properties": { + "description": { + "enabled": true, + "dynamic": false, + "fields": [ + { + "include_in_all": true, + "include_term_vectors": true, + "index": true, + "name": "description", + "store": true, + "type": "text" + } + ] + } + } + } + } + }, + "store": { + "kvStoreName": "mossStore" + } + }, + "sourceType": "couchbase", + "sourceName": "travel-sample", + "sourceUUID": "", + "sourceParams": {}, + "planParams": { + "maxPartitionsPerPIndex": 171, + "numReplicas": 0 + }, + "uuid": "" +} + +
      + +
      + + + Index Definition: geoIndex + + +

      + The following index is used to support the geospatial queries described in + Geospatial Queries. +

      + + { + "type": "fulltext-index", + "name": "geoIndex", + "uuid": "", + "sourceType": "couchbase", + "sourceName": "travel-sample", + "sourceUUID": "", + "planParams": { + "maxPartitionsPerPIndex": 171 + }, + "params": { + "doc_config": { + "mode": "type_field", + "type_field": "type" + }, + "mapping": { + "default_analyzer": "standard", + "default_datetime_parser": "dateTimeOptional", + "default_field": "_all", + "default_mapping": { + "dynamic": true, + "enabled": true, + "properties": { + "geo": { + "dynamic": false, + "enabled": true, + "fields": [ + { + "analyzer": "", + "include_in_all": true, + "include_term_vectors": true, + "index": true, + "name": "geo", + "store": true, + "type": "geopoint" + } + ] + } + } + }, + "default_type": "_default", + "index_dynamic": true, + "store_dynamic": false + }, + "store": { + "kvStoreName": "mossStore" + } + }, + "sourceParams": {} +} +
      + + + +       diff --git a/content/fts/fts-geospatial-queries.dita b/content/fts/fts-geospatial-queries.dita new file mode 100644 index 0000000000..1e841a2758 --- /dev/null +++ b/content/fts/fts-geospatial-queries.dita @@ -0,0 +1,403 @@ + + + + + + Geospatial Queries + + + + Geospatial queries return documents that contain location-data. + + + + +
      + + + Introduction to Geospatial Querying + + +

      + A geospatial query specifies a location, and returns each document that contains + a proximity-match. A location is represented by means of + longitude-latitude coordinate pairs. +

      + +

      + This allows an application, based on the user's input of a particular coordinate, address, or + property name, to derive a list of points of interest that lie within a specified distance + of the user-referenced location; and display these for the user's benefit. +

      + +

      + The location-data provided by a geospatial query can be either of the following: +

      + +
        +
      • + A location, specified as a longitude-latitude coordinate pair; and a distance, in miles. The location + determines the center of a circle whose radius-length is the specified distance. + Documents are returned if they reference a location within the circle. + +

        + +

        +
        • + +
      • + Two longitude-latitude coordinate pairs. These are respectively taken to indicate the top left and + bottom right corners of a bounding box. Documents are returned if they reference a location within the + box. + +

        + +

        +
        • +
      + +

      + To be successful, a geospatial query must reference an index within which the geopoint type mapping + has been applied to the field containing the target longitude-latitude coordinate pair. +

      + +
      + +
      + + + Recognizing Target Data + + +

      + The travel-sample bucket, provided for test and development, + contains multiple documents that specify + locations. For example, those that represent airports, such as airport_1254: +

      + + { + "airportname": "Calais Dunkerque", + "city": "Calais", + "country": "France", + "faa": "CQF", + "geo": { + "alt": 12, + "lat": 50.962097, + "lon": 1.954764 + }, + "icao": "LFAC", + "id": 1254, + "type": "airport", + "tz": "Europe/Paris" +} + +

      + The geo field contains the lon and lat key-value + pairs. Such a parent-field is accessed directly by a geospatial query: the internal long + and lat child-fields are not explicitly specified. Moreover, any other + child-field, such as alt, is ignored. +

      + +

      + For information on installing the travel-sample bucket, see + Install Sample Buckets. +

      + +
      + + +
      + + + Creating a Geospatial Index + + +

      + To be successful, a geospatial query must reference an index that applies the geopoint type mapping to the field + containing the longitude-latitude coordinate pair. + This can be achieved by means of the Couchbase Web Console. Detailed instructions for setting up indexes, and + specifying type mappings, are provided in + Creating Indexes. For + initial experimentation with geospatial querying, the geo field of documents + within the travel-sample + bucket can be specified as a child field of the default type mapping, as follows: +

      + +

      + +

      + +

      + The index so created can also be specified by means of the Couchbase REST API. See + Demonstration Indexes + for the body of the index to be used, and see + Index-Creation + with the REST API for information on using the REST syntax. +

      + +
      + +
      + + + Creating a Query: Radius-Based + + +

      + This section and the following provide examples of the query-bodies required to make geospatial queries with + the Couchbase REST API. + Note that more detailed information on performing queries with the Couchbase REST API can be found in + Searching with the REST API; which + shows how to use the full curl command, and how to incorporate query-bodies into it. +

      + +

      + The following query-body specifies a longitude of -2.235143, and a latitude of 53.482358. + The target-field geo is specified, as is + a distance of 100 miles: this is the radius within which target-locations must reside, + for their documents to be returned. +

      + + { + "from": 0, + "size": 10, + "query": { + "location": { + "lon": -2.235143, + "lat": 53.482358 + }, + "distance": "100mi", + "field": "geo" + }, + "sort": [ + { + "by": "geo_distance", + "field": "geo", + "unit": "mi", + "location": { + "lon": -2.235143, + "lat": 53.482358 + } + } + ] +} + +

      + The query contains a sort object, which specifies that the returned documents + should be ordered in terms of their geo_distance from specified lon and lat + coordinates: these values need not be identical to those specified in the query object. +

      + +

      + A subset of formatted console output might appear as follows: +

      + + . + . + . +"hits": [ + { + "index": "geoIndex_61d8c796ef7f4360_acbbef99", + "id": "landmark_17411", + "score": 1.4045076008239446e-06, + "sort": [ + " \u0001?E#9>N\f\"e" + ] + }, + { + "index": "geoIndex_61d8c796ef7f4360_acbbef99", + "id": "landmark_17409", + "score": 1.4045076008239446e-06, + "sort": [ + " \u0001?O~i*(kD," + ] + }, + { + "index": "geoIndex_61d8c796ef7f4360_acbbef99", + "id": "landmark_17403", + "score": 1.4045076008239446e-06, + "sort": [ + " \u0001?Sg*|/t\u001f\u0002" + ] + }, + { + "index": "geoIndex_61d8c796ef7f4360_acbbef99", + "id": "hotel_17413", + "score": 1.4045076008239446e-06, + "sort": [ + " \u0001?U]S\\.e\u0002_" + ] + }, + . + . + . + +
      + +
      + + + Creating a Query: Box-Based + + +

      + The + following query body forms the top_left corner of a bounding box, specifying a + longitude of -2.235143 and a latitude of 53.482358. This demonstrates + use of an array to specify the coordinate pair: this option can be used interchangeably with that + of using key-value pairs, whenever longitude and latitude are to be specified. + Note that in the array, the lon value must precede the lat. +

      + +

      + The bottom_right of the bounding box is formed by means of key-value pairs; specifying a longitude + of 28.955043 and a latitude of 40.991862. +

      + +

      + If a target data-location falls within the box, + its document is returned. The results are specified to be sorted on name + alone. +

      + + { + "from": 0, + "size": 10, + "query": { + "top_left": [-2.235143, 53.482358], + "bottom_right": { + "lon": 28.955043, + "lat": 40.991862 + }, + "field": "geo" + }, + "sort": [ + "name" + ] +} + +

      + A subset of formatted output might appear as follows: +

      + + . + . + . +"hits": [ + { + "index": "geoIndex_61d8c796ef7f4360_acbbef99", + "id": "landmark_17411", + "score": 1.4045076008239446e-06, + "sort": [ + " \u0001?E#9>N\f\"e" + ] + }, + { + "index": "geoIndex_61d8c796ef7f4360_acbbef99", + "id": "landmark_17409", + "score": 1.4045076008239446e-06, + "sort": [ + " \u0001?O~i*(kD," + ] + }, + { + "index": "geoIndex_61d8c796ef7f4360_acbbef99", + "id": "landmark_17403", + "score": 1.4045076008239446e-06, + "sort": [ + " \u0001?Sg*|/t\u001f\u0002" + ] + }, + { + "index": "geoIndex_61d8c796ef7f4360_acbbef99", + "id": "hotel_17413", + "score": 1.4045076008239446e-06, + "sort": [ + " \u0001?U]S\\.e\u0002_" + ] + }, + . + . + . + +
      + +
      + + + Specifying Distance + + +

      + Multiple unit-types can be used to express distance. These + are listed in the table below, with the strings that specify them in REST queries. +

      + + + + + + + + + + Units + Specify with + + + + + + inches + in or inch + + + + yards + yd or yards + + + + feet + ft or feet + + + + kilometers + km or kilometers + + + + nautical miles + nm or nauticalmiles + + + + millimeters + mm or millimeters + + + + centimeters + cm or centimeters + + + + miles + mi or miles + + + + meters + m or meters + + + + +
      + +
      + + + +       diff --git a/content/fts/fts-performing-searches.dita b/content/fts/fts-performing-searches.dita new file mode 100644 index 0000000000..e81fa3a0e6 --- /dev/null +++ b/content/fts/fts-performing-searches.dita @@ -0,0 +1,139 @@ + + + + + + Performing Searches + + + + Following appropriate preparations, Full Text Searches can be performed in a number of ways. + + + + +
      + + + Preparing for Full Text Searches + + +

      + Before attempting a Full Text Search, you must: +

      + +
        +
      • + Ensure that Couchbase Server has the Search service appropriately enabled. + Enablement for a given node must occur as part of that node's configuration. See + Initializing the Cluster for information. + +

        + +

        + +
        • + +
      • + Create an appropriate index. This can be accomplished by means of the Couchbase Web Console. For + a simple example of index-creation, which allows you to get started quickly, see + Searching from the UI. For + a more detailed explanation of the available options, including index-creation by means of the Couchbase + REST API, see + Creating Indexes. + +

        + +

        + +
        • + +
      + +

      + Note that during index-creation, in support of most query-types, you can select (or create) + and use an analyzer. This + is optional: if you do not specify an analyzer, a default analyzer is provided. + Analyzers can be + created by means of the Couchbase Web Console, during index-creation, as described in + Creating Indexes. Their + functionality and inner components are described in detail in + Understanding Analyzers. +

      + +
      + +
      + + + Performing Full Text Searches + + +

      + Full Text Searches can be performed with: +

      + +
        +
      • + The Couchbase Web Console. This UI can also be used to create indexes and analyzers. See + Searching from the UI, for information. + +

        + +

        + +
        • + +
      • + The Couchbase REST API. See + Searching with the REST API, for information. + See also the page containing REST reference information, + Full Text Search API. + +

        + +

        + +
        • + +
      • + The Couchbase SDK. This supports seven languages, and allows Full Text Searches to be performed with each. See the SDK's + Full Text Search page, for information. + Note also that the + Searching from the SDK page for + the Java SDK provides an extensive code-example that demonstrates multiple options for performing Full Text Searches. (See + immediately below for more information.) + +

        + +

        + +
        • + +
      + +
      + +
      + + + Establishing Demonstration Indexes + + +

      + The Java SDK code-example provided in + Searching with the SDK contains + multiple demonstration calls — each featuring a different query-combination — and makes use of three different index-definitions, related to the + travel-sample bucket: for the code example to run successfully, the three indexes must be + appropriately pre-established. The definitions + are provided in + Demonstration Indexes. Instructions on how to use the Couchbase REST API + to establish the definitions are provided in the section Index-Creation with the REST API, on the page + Creating Indexes. +

      + +
      + + + +       diff --git a/content/fts/fts-queries.dita b/content/fts/fts-queries.dita index acce264dff..0f94248bef 100644 --- a/content/fts/fts-queries.dita +++ b/content/fts/fts-queries.dita @@ -1,329 +1,127 @@ - Queries + + + Understanding Queries + + + + Full Text Search allows queries to be performed on Full Text Indexes. Queries can be performed + by means of the Couchbase Web Console, the Couchbase REST API, or the Couchbase SDK. + + + -

      Queries Using the Web Console

      -

      Once you have created a full text index, you can search it using the Couchbase Server Web - Console (http://your-Couchbase:8091). Go to - Search tab in sidebar. Select your index - row from the table and enter your query using the query string - syntax.

      -

      Queries Using the REST API

      -

      You can also use the REST API to perform queries. The easiest way to get started is to - perform a query using the Web Console. Then, click the "Advanced" - checkbox. You will see the JSON payload to pass to the REST API. Click the nearby - checkbox to get the CURL command. See the REST reference for more - information. - curl -X POST -H "Content-Type: application/json" \ -http://127.0.0.1:8094/api/index/beer-idx/query \ --d '{ - "indexName": "beer-idx", - "size": 10, - "from": 0, - "explain": true, - "highlight": {}, - "query": { - "boost": 1, - "query": "geo.accuracy:rooftop" - }, - "fields": [ - "*" - ], - "ctl": { - "consistency": { - "level": "", - "vectors": {} - }, - "timeout": 0 - } - }'

      -
      Sorting Query Results -

      Unlike previous releases where the search results were always sorted by descending - score, you can now sort search results by any indexed field. If you don't specify a - sort order, the search results are sorted by descending score by default. For more - information on specifying the sort order, see .

      -
      -
      Query Response -

      The response object contains the result of a full text query. It consists of the following objects. For more information, see Response Object Schema.

      - -
      Status
      -
      The status object returns the number of successful and failed pindex queries. This information is useful as you can choose to use partial results depending on your application use case.

      We recommend that you check the status object for failures rather than rely on the HTTP response codes alone. For example,FTS returns an HTTP 200 response in case of pindex failures or timeouts (not consistency timeouts). This is done so that you can choose to accept partial results in an application. However, this also means FTS returns an HTTP 200 response even when ALL pindexes fail.

      -       - -
      Request
      -
      The request object stores a copy of the query that was executed and other details from the request that generated this response object, for example the number of results requested (size), the offset, highlighting, and so on.
      -       - -
      Hits
      -
      Hits returns an array containing the matches for the executed query. The length of the array is equal to or less than the size specified in the request.
      -       - -
      Facets
      -
      Facets return an object that contains aggregated information about the - documents that match a query. There are three types of queries: Numeric - Range Facet, DateTime Range Facets, and Terms Facets. For more information, - see . The facets - object contains the following fields:
        -
      • total - the count of all results returned, regardless of which facet - they are a part of.
        • -
      • count - the count of documents in a specific facet.
        • -
      • other - the count of results that don't fall into any defined - facet.
        • -
      • missing - the count of documents that do not have a value for the - field at all. The total count minus all the individual facet counts - will give you the missing count, which may be 0.
        • -

      The other possible structures vary depending on whether it is a term - facet or a range facet. This means terms have a structure called "term" - with the name, or for a range query, they have some definition of the - range and how to name it. - "facets": { - "abv": { - "field": "abv", - "total": 70, - "missing": 21, - "other": 0, - "numeric_ranges": [ - { - "name": "high", - "min": 7, - "count": 13 - }, - { - "name": "low", - "max": 7, - "count": 57 - } - ] - } -}

      -       - -
      Total_hits
      -
      Total hits represents the total number of matches for this result. It can be any integer starting from 0.
      -       - -
      Max_score
      -
      Max score represents the highest score of all documents for this query.
      -       - -
      Took
      -
      Time taken to complete the query.
      -       -
      - -

      Response Headers

      -

      The response headers can contain the following information: - Response Headers - - - - - - - Code - Example - Valid Return Codes - - - - - Status - HTTP/1.1 200 OK - 200 OK

      400 Bad Request, returned if the query is invalid - due to malformed JSON or invalid consistency - request.

      412 if timeout occurs before the requested - consistency requirements are met.

      -

      For a complete list of status codes and information on how to - interpret them, see Understanding the Query Response Status.

       -       - - Cache-Control - no-cache - - - - Content-Type - application/json; version=1.0.0 - The API version information is included in this field unless - the response is HTTP 400, in which case the response will be - "text/plain: charset=utf-8" - - - Date - Tue, 22 Mar 2016 19:28:57 GMT - Date of the response - - - Transfer-Encoding - chunked - - - - X-Content-Type-Options - nosniff - Value "nosniff" is returned - in case of a bad request (400 or 412) in order to deter driveby - downloads. - - - -

      + +
      -

      Query Counts

      -

      All queries return a result count. To get just the count of documents that match a - particular query without returning documents or ids, execute the query as usual but - specify size "0" to return no results, as in the following - example: - curl -X POST -H "Content-Type: application/json" \ - http://127.0.0.1:8094/api/index/beer-idx/query \ - -d '{ - "indexName": "beer-idx", - "size": 0, - "from": 0, - "explain": true, - "highlight": {}, - "query": { - "boost": 1, - "query": "geo.accuracy:rooftop" - }, - "fields": [ - "*" - ], - "ctl": { - "consistency": { - "level": "", - "vectors": {} - }, - "timeout": 0 - } - }'

      -

      You can get a count of entries in an index overall by using the REST API: - http://localhost:8094/api/index/beer-idx/count

      -
      + + Query-Specification Options + + +

      + Full Text Search allows a range of query options. These include: +

      + +
        + +
      • + Input-text and target-text can be analyzed: this transforms input-text into + token-streams, according to + different specified criteria, so allowing richer and more finely controlled forms of text-matching. + +

        + +

        +
        • + +
      • + The fuzziness of a query can be specified, so that the scope of matches can be constrained to + a particular level of exactitude. A high degree of fuzziness means that a large number of partial + matches may be returned. + +

        + +

        +
        • + +
      • + Multiple queries can be specified for simultaneous processing, with one given a higher boost than + another; so ensuring that its results are returned at the top of the set. + +

        + +

        + +
        • + +
      • + Regular expressions and wildcards can be used in text-specification for search-input. + +

        + +

        +
        • + +
      • + Compound queries can be designed, such that an appropriate conjunction or disjunction of the total + result-set can be returned. + +

        + +

        +
        • + +
      + +

      + All the above options, and others, are explained in detail on the page + Query Types. +

      + +

      + For information on how to execute queries, see + Performing Searches. +

      -
      Types of Queries -

      See for details.

      -
      -
      Search Facets -

      Facets are aggregate information collected on a particular result set. So, you have to - already have a search in mind, and then you collect additional facet information along with - it. All of the facet examples below are for the query "water" on the - beer-sample dataset.

      -

      FTS supports 3 types of facets:

        -
      • Term Facet - A term facet counts up how many of the matching documents have a particular - term in a particular field. Most of the time this only makes sense for relatively low - cardinality fields, like a type or tags. It would not make sense to use it on a unique - field like an ID.
        • -
      • Numeric Range Facet - A numeric range facet works by the user defining their own buckets - (numeric ranges). The facet then counts how many of the matching documents fall into a - particular bucket for a particular field.
        • -
      • Date Range Facet - same as numeric, but on dates instead of numbers. Full text - search and Bleve expect dates to be in the format specified by RFC-3339, which is a specific profile of - ISO-8601 that is more restrictive. For Developer Preview, Date Range - Facets are not supported.
        • -
      Most of the time, when building a term facet you want to use the keyword analyzer. - Otherwise multi-term values get tokenized and the results are not what you - expect.

      -

      Examples

      -
        -
      1. Term Facet - computes facet on the type field which has 2 values: - beer and brewery. - curl -X POST -H "Content-Type: application/json" \ - http://localhost:8094/api/index/bix/query \ - -d '{ - "size": 10, - "query": { - "boost": 1, - "query": "water" - }, - "facets": { - "type": { - "size": 5, - "field": "type" - } - } - }' - The result snippet below only shows the facet section for clarity. Run the curl - command to see the HTTP response containing the full results. - "facets": { - "type": { - "field": "type", - "total": 91, - "missing": 0, - "other": 0, - "terms": [ - { - "term": "beer", - "count": 70 - }, - { - "term": "brewery", - "count": 21 - } - ] - } - }
        2. -
      2. Numeric Range Facet - computes facet on the abv field with 2 - buckets describing high (greater than 7) and - low (less than 7). - curl -X POST -H "Content-Type: application/json" \ - http://localhost:8094/api/index/bix/query \ - -d '{ - "size": 10, - "query": { - "boost": 1, - "query": "water" - }, - "facets": { - "abv": { - "size": 5, - "field": "abv", - "numeric_ranges": [ - { - "name": "high", - "min": 7 - }, - { - "name": "low", - "max": 7 - } - ] - } - } - }' - Results: - facets": { - "abv": { - "field": "abv", - "total": 70, - "missing": 21, - "other": 0, - "numeric_ranges": [ - { - "name": "high", - "min": 7, - "count": 13 - }, - { - "name": "low", - "max": 7, - "count": 57 - } - ] - } - }
        3. -
      -
      +
      + +
      + + + Sorting Results + + +

      + The results returned from a Full Text Search can be sorted: by id, score, field, and more. + Details are provided in + Sorting Query Results. +

      + +
      + +
      + + + Query Response Objects + + +

      + Every Full Text Search query provides a response object. This + contains the query-result; and consists of individual child-objects that + respectively contain status, + a request-copy, the number + of hits (or matches) achieved, and (optionally) facets, which + provide aggregation information. +

      + +

      + For full information, see + Handling Response Objects. +

      + +
      +       diff --git a/content/fts/fts-query-types.dita b/content/fts/fts-query-types.dita index e767d3c766..75269551d0 100644 --- a/content/fts/fts-query-types.dita +++ b/content/fts/fts-query-types.dita @@ -1,23 +1,127 @@ - Types of Queries + + + Query Types + + + + Couchbase Full Text Search supports multiple types of query. + + -

      Different types of Full Text Search queries are available and these query types support - several options for making the precise query you are interested in. This topic describes the - general purpose of each query type along with a sample JSON data that represents the query - type.

      -

      Examples

      -

      The examples below are shown using the REST API query syntax, but you can choose to run these - examples with your - choice of SDK as well. The example query JSON reflects data from the travel-sample - bucket where applicable, and assumes the existence of a Search index called hotel.

      -

      To run the examples using CURL, use the following general syntax: - curl -u Administrator:password -X POST -H "Content-Type: application/json" \ + +

      + + + Introduction to Query Types + + +

      + Full Text Search allows text-data to be queried. Multiple options are provided for + ensuring the right kinds of match. This page describes the + purpose of each query-type, and provides sample JSON objects that indicate how queries + can be constructed. +

      + +

      + Available query-types include: +

      + +
        + +
      • + Simple Queries: Accept input-text in the form of words and phrases, and attempt to find + matches across bodies of text that have been indexed. Analyzers are applied + to both input and target, potentially to strip out unnecessary characters, reduce words to + the basic stems on which matching should occur, handle punctuation, and more. Additionally, + match accuracy-levels can be specified; and + multiple queries can be expressed together, with their respective priorities boosted, + (to ensure their results' prominence in the eventual result-set). + +

        + +

        +
        • + +
      • + Compound Queries: Accept multiple queries simultaneously, and return either the + conjunction of results from the result-sets, or a disjunction. + +

        + +

        +
        • + + +
      • + Range Queries: Accept ranges for dates and numbers, and return documents that contain + values within those ranges. + +

        + +

        +
        • + +
      • + Query String Queries: Accept query strings, which express query-requirements + in a special syntax. + +

        + +

        +
        • + +
      • + Geospatial Queries: Accept longitude-latitude coordinate pairs, in order + to return documents that specify a geographical location. +
        • + +
      • + Non-Analytic Queries: Accept words and phrases on which exact matches only are returned. No + analysis is performed. + +

        + +

        +
        • + +
      • + Special Queries: For testing purposes, return either all of the documents in an index, + or none. + +

        + +

        +
        • + + +
      + +

      + These query-types are explained in greater detail below. Examples are provided, using + the Couchbase REST API query-syntax. (Note that Full Text Search can also + be performed with the Couchbase Web Console and the Couchbase SDK.) The JSON data refers to + the travel-sample + bucket, and assumes the existence of a Full Text Index named hotel. +

      + +

      + To run the examples using curl, use the following syntax: +

      + + curl -u Administrator:password -X POST -H "Content-Type: application/json" \ -d '{your query in JSON here...}' \ - http://localhost:8094/api/index/index_name/query

      -

      Note that the queries below only show a portion of the query relevant to the query type being - described. Wrap the query details in the following basic configuration. + http://localhost:8094/api/index/index_name/query + + +

      + Note that the examples below show only the JSON fragments that constitute non-generic parts of the queries they + describe. For actual use in a Full Text Search, these JSON fragments should be wrapped in the following generic + configuration: +

      + { "explain": false, "fields": [ @@ -25,277 +129,751 @@ ], "highlight": {}, "query":{ your_query_details_here } -}

      -
      Simple Queries +} + +

      + For more information on using the REST API to perform queries, + see + Searching with the REST API. +

      + +
      + +
      + + + Simple Queries + +
      + -
      Match Query
      -
      A match query analyzes the input text and uses that analyzed text to query the index. - Options include specifying a analyzer, performing fuzzy matching or adding in a prefix - match. An attempt is made to use the same analyzer that was used when the field was indexed. -

      The match query can optionally perform fuzzy matching. If the fuzziness parameter - is set to a non-zero integer the analyzed text will be matched with the specified - level of fuzziness. Also, the prefix_length parameter can be used - to specify that the term is also required to have the same prefix of the specified - length.

      { + +
      + Match Query +
      + +
      + A match query analyzes input text, and uses the results to query an index. + Options include specifying an analyzer, performing a fuzzy match, and performing + a prefix match. By default, the analyzer used for the input-text is that previously used on + the target-text, during index-creation. For information on analyzers, see + Understanding Analyzers. + +

      + When fuzzy matching is used, if the single parameter + is set to a non-zero integer, the analyzed text is matched with a corresponding + level of fuzziness. +

      + +

      + When a prefix match is used, the prefix_length parameter specifies + that for a match to occur, a prefix of specified length must be shared by the + input-term and the target text-element. +

      + +

      + The following JSON object demonstrates specification of a match query: +

      + + { "match": "location hostel", "field": "reviews.content", "analyzer": "standard", "fuzziness": 3, "prefix_length": 4 -}
      +} + +

      + A match query is also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + + +       + -
      Match Phrase Query
      -
      The input text is analyzed and a phrase query is built with the terms resulting from - the analysis. This type of query searches for terms occurring in the specified positions - and offsets. This depends on term vectors, which are consulted to determine phrase - distance.

      Although this is not an exact match, it is usually the search behavior - users want.

      For example, a match phrase query for "location for functions" might - match "locate the function" if the standard en analyzer is used. This - is because the stemmer tokenizes "location" and "locate" to "locati", and "functions" - and "function" to "function." Stop word removal will remove "for" and "the" but the - distance between the tokens will be the same. -

      { + +
      + Match Phrase Query +
      + +
      + The input text is analyzed, and a phrase query is built with the terms resulting from + the analysis. This type of query searches for terms in the target that occur in the positions + and offsets indicated by the input: this depends on term vectors, which must have been included in the + creation of the index used for the search. + +

      + For example, a match phrase query for location for functions is matched + with locate the function, if the standard analyzer is used: this analyzer uses a + stemmer, which tokenizes location and locate to locat, and + reduces functions + and function to function. Additionally, the analyzer employs + stop removal, which removes small and less significant words from input and target text, so that + matches are attempted on only the more significant elements of vocabulary: in this case for and + the are removed. Following this processing, the tokens locat and + function are recognized as common to both input and target; and also as being both + in the same sequence + as, + and at the same distance from one another; and therefore a match is made. +

      + + { "match_phrase": "very nice", "field": "reviews.content" -}
      +} + +

      + A match phrase query is also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + + +       + -
      Fuzzy Query
      -
      A fuzzy query is a term query that matches terms within a specified edit distance - (Levenshtein distance). Also, you can optionally specify that the term must have a - matching prefix of the specified - length.{ + +
      + Fuzzy Query +
      + +
      + A fuzzy query matches terms within a specified edit (or Levenshtein) distance: meaning + that terms are considered to match when they are to a specified degree similar, rather than exact. + A common prefix of a stated length may be also specified as a requirement for matching. + +

      + Fuzziness is specified by means of a single integer. For example: +

      + + { "term": "interest", "field": "reviews.content", "fuzziness": 2 -}
      +} + +

      + Fuzzinesss is demonstrated by means of the Java SDK, in the context of the term query + (see below), in + Searching with the SDK. Note + that two such queries are specified, with the difference in fuzziness between them resulting in different forms of + match, and different sizes of result-sets. +

      + + + +       + -
      Prefix Query
      -
      The prefix query finds documents containing terms that start with the provided prefix. + +
      + Prefix Query +
      + +
      + A prefix query finds documents containing terms that start with the specified prefix. + { "prefix": "inter", "field": "reviews.content" -}
      +} + + +       + -
      Regexp Query
      -
      Regexp query finds documents containing terms that match the specified regular - expression. { + +
      + Regexp Query +
      + +
      + A regexp query finds documents containing terms that match the specified regular expression. + + { "regexp": "inter.+", "field": "reviews.content" -}
      +} + +

      + A regexp query is also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + + +       + -
      Wildcard Query
      -
      The wildcard query uses a wildcard expression to search within individual terms for - matches. Wildcard expressions can be any single character (?) or zero to many characters - (*). Wildcard expressions can appear in the middle or end of a term but not at the - beginning of the search term. + +
      + Wildcard Query +
      + +
      + A wildcard query uses a wildcard expression, to search within individual terms for + matches. Wildcard expressions can be any single character (?) or zero to many characters + (*). Wildcard expressions can appear in the middle or end of a term, but not at the + beginning. + { "wildcard": "inter*", "field": "reviews.content" -}
      +} + +

      + A wildcard query is also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + + +       + -
      Boolean Field Query
      -
      The Boolean field query searches a field that contains Boolean true or false values. A - Boolean field query searches the actual content of the field, and should not be confused - with the Boolean - queries (described in the section on compound queries) that modify whether a + +
      + Boolean Field Query +
      + +
      + A boolean field query searches fields that contain boolean true or + false values. A + boolean field query searches the actual content of the field, and should not be confused + with the + boolean queries + (described below, in the section on compound queries) that modify whether a query must, should, or must not be present. + { "bool": true, "field": "free_breakfast" -}
      +} + + +       -
      -
      Compound Queries + + +
      + +
      + + + Compound Queries + +
      + -
      Conjunction Query (AND)
      -
      The conjunction query is a compound query. The result documents must satisfy all of + +
      + Conjunction Query (AND) +
      + +
      + A conjunction query contains multiple child queries. Its result documents must satisfy all of the child queries. + { "conjuncts":[ {"field":"reviews.content", "match": "location"}, {"field":"free_breakfast", "bool": true} ] -}
      +} + +

      + A conjunction query is also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + + +       + -
      Disjunction Query (OR)
      -
      The disjunction query is a compound query. The result documents must satisfy a - configurable min number of child queries. By default this - min is set to 1. + +
      + Disjunction Query (OR) +
      + +
      + A disjunction query contains multiple child queries. Its result documents must satisfy a + configurable min number of child queries. By default this + min is set to 1. For example, if three child queries — A, B, and C — are + specified, a min of 1 specifies that the result documents should be those returned + uniquely for A (with all returned uniquely for B and C, and all returned commonly for A, B, and C, omitted). + { "disjuncts":[ {"field":"reviews.content", "match": "location"}, {"field":"free_breakfast", "bool": true} ] -}
      +} + +

      + A disjunction query is also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + + +       + -
      Boolean Query
      -
      The boolean query is a useful combination of conjunction and disjunction queries. A - boolean query takes three lists of queries:
        -
      • must - result documents must satisfy all of these queries.
        • -
      • should - result documents should satisfy these queries.
        • -
      • must not - result documents must not satisfy any of these queries.
        • -
      { +
      + Boolean Query +
      + +
      + A boolean query is a combination of conjunction and disjunction queries. A + boolean query takes three lists of queries: + +
        + +
      • + must: Result documents must satisfy all of these queries. +
        • + +
      • + should: Result documents should satisfy these queries. +
        • + +
      • + must not: Result documents must not satisfy any of these queries. +
        • +
      + + { "must": { "conjuncts":[{"field":"reviews.content", "match": "location"}]}, "must_not": { "disjuncts": [{"field":"free_breakfast", "bool": false}]}, "should": { "disjuncts": [{"field":"free_breakfast", "bool": true}]} -}
       +} + + + + -
      Doc ID Query
      -
      The doc ID query returns the indexed document or documents among the specified set. This - is typically used in conjunction queries to restrict the scope of other queries’ output. - { "ids": [ "hotel_10158", "hotel_10159" ] }
      + +
      + Doc ID Query +
      + +
      + A doc ID query returns the indexed document or documents among the specified set. This + is typically used in conjunction queries, to restrict the scope of other queries’ output. + + { "ids": [ "hotel_10158", "hotel_10159" ] } + + +

      + A doc ID Query is demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + +
      +       +
      +
      -
      Range Queries + +
      + + + Range Queries + +
      + -
      Date Range Query
      -
      The date range query finds documents containing a date value in the specified field - within the specified range. Full text search and Bleve expect dates to be in the format - specified by RFC-3339, which is a specific profile of ISO-8601 that is more - restrictive. Define the endpoints using the fields start and - end. You can omit one endpoint, but not both. The + +
      + Date Range Query +
      + +
      + A date range query finds documents containing a date value, in the specified field + within the specified range. Dates should be in the format + specified by RFC-3339, + which is a specific profile of ISO-8601. Define the endpoints using the fields start and + end. One endpoint can be omitted, but not both. The inclusiveStart and inclusiveEnd properties - in the query JSON control whether or not the end points are included or excluded. + in the query JSON control whether or not the endpoints are included or excluded. + { "start": "2001-10-09T10:20:30-08:00", "end": "2016-10-31", "inclusive_start": false, "inclusive_end": false, "field": "review_date" -}
      +} + +       + -
      Numeric Range Query
      -
      The numeric range query finds documents containing a numeric value in the specified + +
      + Numeric Range Query +
      + +
      + A numeric range query finds documents containing a numeric value in the specified field within the specified range. Define the endpoints using the fields min and max. You can omit one endpoint, but not both. The inclusiveMin and inclusiveMax - properties control whether or not the end points are included or excluded. By default, + properties control whether or not the endpoints are included or excluded. By default, min is inclusive and max is exclusive. + { "min": 100, "max": 1000, "inclusive_min": false, "inclusive_max": false, "field": "id" -}
      +} + +

      + A numeric range Query is also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + + + + + + +
      -
      Query String Query -

      Also known as the String Query, the query string query allows humans to describe complex queries using a simple syntax. When you search a full text index with the Web Console, you use this query syntax.

      Certain queries that are supported by FTS are not yet supported by the query string syntax. This includes wildcards, regexp, and date range queries. + +
      + + + Query String Query + + +

      + A query string can be used, to express a given query by means of a special syntax. +

      + { "query": "+nice +view" } -
      - -
      Match
      -
      A term without any other syntax is interpreted as a match query for the term in the default - field. The default field is _all unless overridden in the index mapping. -

      Example: water performs a Match Query for the term "water".

      -       - -
      Match Phrases
      -
      Placing the search terms in quotes performs a match phrase query. This is not an exact match, see Match Phrases Query for more information.

      Example: "light beer" performs a Match Phrases Query for the phrase "light beer".

      -       - -
      Field Scoping
      -
      You can qualify the field for these searches by prefixing them with the name of the field separated by a colon. -

      Example: description:water performs a Match Query for the term "water", in the description field.

      -       - -
      Required, Optional, and Exclusion
      -
      When your query string includes multiple items, by default these are placed into the SHOULD - clause of a Boolean Query.

      You can change this by prefixing your items with a + or ‘-’. * ‘+’ Prefixing with plus places that item in the MUST portion of the boolean query. * ‘-’ Prefixing with a minus places that item in the MUST NOT portion of the boolean query.

      -

      Example: +description:water -light beer will perform a boolean query that MUST satisfy the match query for the term "water" in the description field, MUST NOT satisfy the match query for the term "light" in the default field, and SHOULD satisfy the match query for the term "beer" in the default field. Result documents satisfying the SHOULD clause will score higher than those that do not.

      -       - -
      Boosting
      -
      You can influence the relative importance of the clauses by suffixing clauses with the ^ operator followed by a number.

      Example: description:water name:water^5 will perform match queries for "water" in both the name and description fields, but documents having the term in the name field will score higher.

      -       - -
      Numeric Ranges
      -
      You can perform numeric ranges by using the >, >=, <, and <= operators, followed by a numeric value.

      Example: abv:>10 will perform an Numeric Range Query on the abv field for values greater than ten.

      -       -
      + +

      + A query string Query is demonstrated by means of the Java SDK, in + Searching with the SDK. + Note also that the Full Text Searches conducted with the Couchbase Web Console themselves + use query strings. (See + Searching from the UI.) +

      + +

      + Certain queries supported by FTS are not yet supported by the query string syntax. + This includes wildcards, regexp, and date range queries. +

      + +

      + Using the query string syntax, the following query types can be performed: +

      + +
      + + + +
      + Match +
      + +
      + A term without any other syntax is interpreted as a match query for the term in the default + field. The default field is _all. + +

      + For example, water performs a + Match Query + for the term water. +

      + +
      + +       + + + +
      + Match Phrases +
      + +
      + Placing the search terms in quotes performs a match phrase query. For example: + light beer performs a + match phrase query + for the phrase light beer. +
      + +       + + + +
      + Field Scoping +
      + +
      + The specified field in which to search can be specified by prefixing the term with a field-name, separated by a colon. + For example, description:water + performs a match query for the + term water, in the description field. +
      + +       + + + +
      + Required, Optional, and Exclusion +
      + +
      + + When a query string includes multiple items, by default these are placed into the SHOULD + clause of a + boolean query. This can be adjusted, by prefixing items + with either + or -. + Prefixing with + places that item in the MUST portion of the boolean query. Prefixing + with - places that item in the MUST NOT portion of the boolean query. + +

      + For example, +description:water -light beer performs a boolean query that MUST satisfy the + match query for the term water in the description field, MUST NOT satisfy + the match query + for the term light in the default field, and SHOULD satisfy the match query + for the term beer in the default field. Result documents satisfying the + SHOULD clause score higher than those that do not. +

      + +
      + +       + + + +
      + Boosting +
      + +
      + + When multiple query-clauses are specified, the relative importance a given clause can be specified + by suffixing it with the ^ operator, + followed by a number. For example, description:water name:water^5 performs Match Queries + for water in both the name and description fields, + but documents having the term in the name field score higher. + +
      + +       + + + +
      + Numeric Ranges +
      + +
      + + Numeric ranges can be specified with the >, >=, <, and <= operators, each followed by a + numeric value. For example, abv:>10 performs a numeric range query on + the abv field, for values greater than 10. + +

      + A numeric range query is demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + +
      + +       + +
      +
      -
      Special Queries -

      Special queries are usually used in combination with other queries or used to test the systems.

      + +
      + + + Non-Analytic Queries + + +

      + Term and Phrase queries support no analysis on + their inputs. This means that only exact matches are returned. +

      + +

      + In most cases, given the benefits of using analyzers, use of match and match phrase queries + is preferable to that of term and phrase. For information on analyzers, see + Understanding Analyzers. +

      + +
      + -
      Match All Query
      -
      Matches all documents in the index. Be aware that this query will match all documents - that were indexed even if they have no terms in the index. For example, if a custom map - is created on the travel-sample bucket that indexes documents of type="zucchini", the - match all query returns all document IDs in the bucket even though there are no - documents of type="zucchini". This can be very counter-intuitive. - { "match_all": {} }
      + +
      + Term Query +
      + +
      + A term query is the simplest possible query. It performs an exact + match in the index for the provided term. + + { + "term": "locate", + "field": "reviews.content" +} + +

      + Term queries are also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + +
      +       + -
      Match None Query
      -
      Matches no documents in the index. { "match_none": {} }
      + +
      + Phrase Query +
      + +
      + A phrase query searches for terms occurring in the specified position and offsets. + It performs an exact term-match for all the phrase-constituents, + without using an analyzer. + + { + "terms": ["nice", "view"], + "field": "reviews.content" +} + +

      + A phrase query is also demonstrated by means of the Java SDK, in + Searching with the SDK. +

      + +
      +       +
      + +
      + +
      + + + Geospatial Queries + + +

      + Geospatial queries return documents that each specify a geographical location. Each query contains either: +

      + +
        +
      • + A single longitude-latitude coordinate pair; and a distance value, in miles, which + determines a radius measured from the location specified by the coordinate pair. Documents are returned if they + specify (by means of a longitude-latitude coordinate pair) a location that lies within the radius. + +

        + +

        +
        • + +
      • + Two longitude-latitude coordinate pairs. These are respectively taken to indicate the upper left and + lower right corners of a bounding box. Documents are returned if they specify a location that lies within the + bounding box. + +

        + +

        +
        • +
      + +

      + A geospatial query must be applied to an index that applies the geopoint type mapping + to the document-field that + contains the target longitude-latitude coordinate pair. +

      + +

      + More detailed information is provided in + Geospatial Queries. +

      -
      - Queries without Analysis -

      Term and Phrase queries do not perform any analysis on - their inputs. This is useful for debugging exactly what is in an index, and in certain other - special cases, but it is often used by accident. A term search is often - used in a compound query to include or exclude certain documents where you can check an - exact value.

      -

      Typically, users want match and phrase match queries - instead. In most full text queries, an analyzer is used on both the text to be indexed and - the query terms. Because the terms in the index have usually been run through an analyzer - and a stemmer, they often no longer look like normal language. For example, after analysis - the terms in an index might look like "natio" or "beaut" instead of "National" or - "beauties". End users of search systems never see the stored terms but, because their - queries are also run through the same analyzer that was used to build the index, the results - match.

      -

      + +
      + + + Special Queries + + +

      + Special queries are usually employed either in combination with other queries, or to test the system. +

      +
      + -
      Term Query
      -
      A term query is the simplest possible query. It performs an exact - match in the index for the provided term. Most of the time users should use a Match - Query instead.

      For example, a term query for "lovingli" would match documents with - the term "lovingly" if they were indexed with the "en" analyzer. A - term query for the "lovingly" won’t match any documents that were indexed with the - "en" analyzer. -

      { - "term": "locate", - "field": "reviews.content" -}
      + +
      + Match All Query +
      + +
      + Matches all documents in an index, + irrespective of terms. For example, if an index + is created on the travel-sample bucket for documents of type zucchini, the + match all query returns all document IDs from the travel-sample bucket, even though + the bucket + contains no documents of type zucchini. + + { "match_all": {} }
      +       + -
      Phrase Query
      -
      A phrase query searches for terms occurring in the specified position and offsets. -

      The phrase query performs an exact term match for all the phrase constituents - without using an analyzer. If you want the phrase to be analyzed, consider using the - Match Phrase Query - instead.

      { - "terms": ["nice", "view"], - "field": "reviews.content" -}
      + +
      + Match None Query +
      + +
      + Matches no documents in the index. + + { "match_none": {} } + +
      +       +
      +
      + + diff --git a/content/fts/fts-response-object-schema.dita b/content/fts/fts-response-object-schema.dita index e8cf418ae7..cfecc52270 100644 --- a/content/fts/fts-response-object-schema.dita +++ b/content/fts/fts-response-object-schema.dita @@ -1,15 +1,56 @@ - Response Object Schema + + + Handling Response Objects + + + + Full Text Search provides a response object, which contains detailed information on + the results of the search. Prior to the search, + a facet can be add to the response object, and + information on aggregations thereby returned. + + -

      -
      Status -

      The status object returns the number of successful and failed pindex queries. This - information is useful as you can choose to use partial results depending on your application - use case. For example, you may prefer to show an incomplete list of hits for a catalog - search than return an error message if some pindexes are unresponsive.

      We recommend that you check the status object for failures rather than rely on the HTTP response codes alone. For example,FTS returns an HTTP 200 response in case of pindex failures or timeouts (not consistency timeouts). This is done so that you can choose to accept partial results in an application. However, this also means FTS returns an HTTP 200 response even when ALL pindexes fail.

      -

      Refer to the following table for more information about the possible status messages. + + + Response-Object Structure + + +

      + A Full Text Search Response Object is itself composed of multiple child-objects, each of + which provides important + information. Each child-object is listed and explained below. +

      + + + +
      + + + Status + + +

      + The status object includes the number of successful and failed pindex queries. +

      + +

      + We recommend that the status object be checked for failures, as a preferred alternative to relying on HTTP + response codes alone. For example,FTS returns an HTTP 200 response in case of pindex failures + or timeouts (not consistency timeouts). This is done so that you can choose to accept partial + results in an application. However, this also means FTS returns an HTTP 200 response even when + ALL pindexes fail. +

      + +

      + Refer to the following table for more information about the possible status messages. + +

      Response Object Status Values @@ -46,7 +87,7 @@
      Total
      The total field returns an integer representing the total number of pindexes queried. This value varies depending on how your index is configured, the - platform you're running on, and whether you are querying an index or an index alias. + platform you are running on, and whether you are querying an index or an index alias. Each index has a setting for number of vBuckets per pindex. For example, when running on a cluster with 1024 vBuckets and with an index setting of 171 vBuckets per pindex, the total number of pindexes is 6. When you query an This section explains the contents of a typical query response status.

      Request -

      The request object stores a copy of the query that was executed and other details from the request that generated this response object, for example the number of results requested (size), the offset, highlighting, and so on.

      +

      The request object stores a copy of the query that was executed and other details from the request that generated + this response object: for example the number of results requested (size), the offset, highlighting, and so on.

      Query
      Represents the actual query that was executed.
      @@ -127,8 +169,7 @@
      Facets
      This object indicates whether faceting was requested and when requested, returns the - facets. The section on - provides more information on the possible values of the facet request. + facets. See below for detailed information on facets. "facets": { "type": { "size": 5, @@ -139,12 +180,20 @@
      Explain
      This Boolean field when set to true prints a verbose response with full scoring - information.
      + information.
      -
      Hits -

      Hits returns an array containing the matches for the executed query. The length of the array is equal to or less than the size specified in the request.

      + +
      + + Hits + + +

      + Hits returns an array containing the matches for the executed query. The length of the array is equal to or less than the size specified in the request. +

      +
      Index
      @@ -196,6 +245,7 @@ },       +
      Fragments
      These objects, also known as snippets, contain field names that contain an array of @@ -216,23 +266,302 @@
      -
      Facets -

      Facets return an object that contains aggregated information about the documents that match - a query. There are three types of queries: Numeric Range Facet, DateTime Range Facets, and - Terms Facets. For more information on facets, see .

      -
      +
      Total_hits

      Total hits represents the total number of matches for this result. It can be any integer starting from 0.

      +
      Max_score

      Max score represents the highest score of all documents for this query.

      -
      Took + +
      + Took +

      Time taken to complete the query. "total_hits": 56, "max_score": 0.8041158525040355, "took": 1449005,

      + + +
      + + + Response Headers + + +

      + Response headers can contain the following information: +

      + +
      + Response Headers + + + + + + + Code + Example + Valid Return Codes + + + + + Status + HTTP/1.1 200 OK + 200 OK

      400 Bad Request, returned if the query is invalid + due to malformed JSON or invalid consistency + request.

      412 if timeout occurs before the requested + consistency requirements are met.

      +       +       + + Cache-Control + no-cache + + + + Content-Type + application/json; version=1.0.0 + The API version information is included in this field unless + the response is HTTP 400, in which case the response will be + "text/plain: charset=utf-8" + + + Date + Tue, 22 Mar 2016 19:28:57 GMT + Date of the response + + + Transfer-Encoding + chunked + + + + X-Content-Type-Options + nosniff + Value "nosniff" is returned + in case of a bad request (400 or 412) in order to deter driveby + downloads. + + + +
      + +

      + +
      + + + Query Counts + + +

      + All queries return a result count. To get just the count of documents that match a + particular query without returning documents or ids, execute the query as usual but + specify size "0" to return no results, as in the following + example: +

      + + curl -X POST -H "Content-Type: application/json" \ +http://127.0.0.1:8094/api/index/beer-idx/query -d \ +'{ + "indexName": "beer-idx", + "size": 0, + "from": 0, + "explain": true, + "highlight": {}, + "query": { + "boost": 1, + "query": "geo.accuracy:rooftop" + }, + "fields": [ + "*" + ], + "ctl": { + "consistency": { + "level": "", + "vectors": {} + }, + "timeout": 0 + } +}' + +

      + You can get a count of entries in an index overall by using the REST API: +

      + + http://localhost:8094/api/index/beer-idx/count + +
      + +
      + + + Search Facets + + +

      + Facets are aggregate information collected on a particular result set. So, you have to + already have a search in mind, and then you collect additional facet information along with + it. All of the facet examples below are for the query "water" on the + beer-sample dataset. +

      + +

      + FTS supports 3 types of facet: +

      + +
        +
      • + Term Facet - A term facet counts up how many of the matching documents have a particular + term in a particular field. Most of the time this only makes sense for relatively low + cardinality fields, like a type or tags. It would not make sense to use it on a unique + field like an ID. +
        • + +
      • + Numeric Range Facet - A numeric range facet works by the user defining their own buckets + (numeric ranges). The facet then counts how many of the matching documents fall into a + particular bucket for a particular field. +
        • + +
      • + Date Range Facet - same as numeric, but on dates instead of numbers. Full text + search and Bleve expect dates to be in the format specified by + RFC-3339, + which is a specific profile of + ISO-8601 that is more restrictive. + +
        • + +
      + + Most of the time, when building a term facet you want to use the keyword analyzer. + Otherwise multi-term values get tokenized and the results are not what you + expect. + + +

      + Examples +

      + +
        + +
      • + Term Facet - computes facet on the type field which has 2 values: beer and brewery. + + curl -X POST -H "Content-Type: application/json" \ +http://localhost:8094/api/index/bix/query -d \ +'{ + "size": 10, + "query": { + "boost": 1, + "query": "water" + }, + "facets": { + "type": { + "size": 5, + "field": "type" + } + } +}' + +

        + The result snippet below only shows the facet section for clarity. Run the curl + command to see the HTTP response containing the full results. +

        + + "facets": { + "type": { + "field": "type", + "total": 91, + "missing": 0, + "other": 0, + "terms": [ + { + "term": "beer", + "count": 70 + }, + { + "term": "brewery", + "count": 21 + } + ] + } +} + +
        • + +
      • + Numeric Range Facet - computes facet on the abv field with 2 + buckets describing high (greater than 7) and + low (less than 7). + + curl -X POST -H "Content-Type: application/json" \ +http://localhost:8094/api/index/bix/query -d \ +'{ + "size": 10, + "query": { + "boost": 1, + "query": "water" + }, + "facets": { + "abv": { + "size": 5, + "field": "abv", + "numeric_ranges": [ + { + "name": "high", + "min": 7 + }, + { + "name": "low", + "max": 7 + } + ] + } + } +}' + +

        + Results: +

        + + facets": { + "abv": { + "field": "abv", + "total": 70, + "missing": 21, + "other": 0, + "numeric_ranges": [ + { + "name": "high", + "min": 7, + "count": 13 + }, + { + "name": "low", + "max": 7, + "count": 57 + } + ] + } +} + +
        • + +
      + +
      +       diff --git a/content/fts/fts-searching-from-the-ui.dita b/content/fts/fts-searching-from-the-ui.dita new file mode 100644 index 0000000000..4e3eaa1cbe --- /dev/null +++ b/content/fts/fts-searching-from-the-ui.dita @@ -0,0 +1,250 @@ + + + + + + Searching from the UI + + + + Full Text Search can be performed from the Couchbase Web Console. + + + + +
      + + + Access the Full Text Search User Interface + + +

      + The user interface for Full Text Search is provided by the Couchbase Web Console. In order to + proceed, you must have permission to log into the console, create indexes, and perform searches. For + information on Role-Based Access Control, see + Authorization. +

      + +

      + The example provided in this section assumes that you have also loaded the + travel-sample + bucket: you will perform your Full Text Search operations on the data in this bucket. For instructions on how + to load this sample bucket, see + Install Sample Buckets. +

      + +

      + Once you have the appropriate credentials, and have loaded the travel-sample bucket, + access the Couchbase Web Console by typing + http://localhost:8091 into the address-field at the top of your browser, and then + hitting return. The login screen appears as follows: +

      + +

      + +

      + +

      + Enter your username and password, and left-click on the Sign In button. The Couchbase Web + Console now appears: +

      + +

      + +

      + +

      + To access the Full Text Search screen, left-click on the Search tab, in the navigation + bar at the left-hand side: +

      + +

      + +

      + +

      + The Full Text Search screen now appears, as follows: +

      + +

      + +

      + +

      + The console contains areas for the display of indexes and aliases: but both are empty, since none + has yet been created. +

      + +
      + +
      + + Create an Index + + +

      + To create an index, left-click on the Add Index button, towards the right-hand side: +

      + +

      + +

      + +

      + The Add Index screen appears: +

      + +

      + +

      + +

      + To define a basic index on which Full Text Search can be performed, begin by entering a unique name + for the index into the Name field, at the upper-left: for example, travel-sample-index. + (Note that only alphanumeric characters, hyphens, and underscores are allowed for index names. Note also that the + first character of the name must be an alphabetic character.) + Then, use the pull-down menu provided for + the Bucket field, at the upper-right, to specify the travel-sample bucket: +

      + +

      + +

      + +

      + This is all you need to specify, in order to create a basic index for test and development. No further configuration is + required. Note, however, + that such default indexing is + not recommended for production environments, since it creates indexes that may be unnecessarily large, and therefore + insufficiently performant. To review the wide range of available options + for creating indexes appropriate for production environments, see + Creating Indexes. +

      + +

      + To save your index, left-click on the Create Index button, near the bottom of the screen: +

      + +

      + +

      + +

      + At this point, you are returned to the Full Text Search screen. A row now appears, in the Full Text Indexes + panel, for the index you have created. When left-clicked on, the row opens as follows: +

      + +

      + +

      + +

      + Note the percentage figure: this appears under the indexing progress column, and is incremented in correspondence + with the build-progress of the index. When 100% is reached, the build is complete, and the index ready for use. +

      + +

      + Once the + the new index has been built, it supports Full Text Searches performed by all available means: the Console UI, + the Couchbase REST API, and the Couchbase SDK. +

      + +
      + +
      + + + Perform a Query + + +

      + To perform a query, simply type a term into the interactive text-field that appears to the left of the Search button on the row + for the index you have created. For example, restaurant. Then, left-click on the Search button: +

      + +

      + +

      + +

      + A Search Results page now appears, featuring documents that contain the specified term: +

      + +

      + +

      + +

      + By left-clicking on any of the displayed document IDs, you bring up a display that features the entire contents of the document. +

      + +
      + +
      + + Advanced Settings and Other Features + + +

      + On the Search Results page, to the immediate right of the Search button, at the top of the screen, appears + the show advanced query settings checkbox. Check this to display the advanced settings: +

      + +

      + +

      + +

      + Three interactive text-fields now appear underneath the Search panel: Timeout (msecs), + Consistency Level, and Consistency Vector. Additionally, the JSON for Query Request + panel displays the submitted query in JSON format. Note the show command-line curl example checbox, + which when checked, adds to the initial JSON display, to form a completed curl command: +

      + +

      + +

      + +

      + This example can be copied by means of the Copy to Clipboard + button, pasted (for example) into a standard console-window, and executed against the prompt. This feature + therefore + provides a useful means of extending experiments initially performed with the UI into a + subsequent console-based, script-based, or program-based context. (Note, however, that the addition + of credentials for authentication are required for execution of the statement outside the context of + the current session within Couchbase Web Console. See + Searching with the REST API + for an example.) +

      + +

      + Note also the Show Scoring checkbox that appears prior to the entries in the Results for travel-sample-index + panel. When this is checked, scores for each document in the list are provided. For example: +

      + +

      + +

      + +

      + Finally, note the full text query syntax help link that now appears under the Search interactive text-field: +

      + +

      + +

      + +

      + This link takes the user to a + page of + information on Query String Full Text Search queries. Such queries can be specified in the Search interactive + text-field, thereby allowing searches of considerable complexity to be accomplished within the Couchbase Web Console. Additional + information on Query String searches is also provided in + Query Types. +

      + +
      + + + +       diff --git a/content/fts/fts-searching-with-the-rest-api.dita b/content/fts/fts-searching-with-the-rest-api.dita new file mode 100644 index 0000000000..660914ada4 --- /dev/null +++ b/content/fts/fts-searching-with-the-rest-api.dita @@ -0,0 +1,201 @@ + + + + + + Searching with the REST API + + + + Full Text Search can be performed by means of the Couchbase REST API. + + + + +
      + + + Performing a Full Text Search with REST at the Command-Line + + +

      + Full Text Search can be performed with the Couchbase + REST API; at the command-line, by means of the curl utility. +

      + +

      + The syntactic elements for a curl-based Full Text Search can be obtained from the Couchbase + Web Console: the console allows searches performed via the UI to be translated dynamically into + curl examples. To demonstrate this, follow the procedures for + accessing the Full Text Search screen, within the Couchbase Web Console, and for performing + a simple search; as described in + Searching from the UI. + Then, left-click on the show advanced query settings checkbox, at the right-hand side of the + Search button: +

      + +

      + +

      + +

      + The JSON for Query Request + panel displays the submitted query in JSON format. Note the show command-line curl example checkbox, + which when checked, adds to the content of the initial JSON display, to form a completed curl command: +

      + +

      + +

      + +

      + This example can be copied by means of the Copy to Clipboard + button, pasted into (for example) a standard console-window, and executed against the prompt. This feature + therefore + provides a useful means of extending experiments initially performed with the UI into a + subsequent console-based, script-based, or program-based context. Note, however, that authentication + is required for the call to be successful from any context outside the current Couchbase Web Console + session. Additionally, familiarity with query strings + should be acquired, for the creation of more complex queries. +

      +
      + + + +
      + + + Query Strings and Authentication + + +

      + A Query String combines standard alphanumeric characters with syntactic elements, in order + to specify complex queries in ASCII form. Query Strings can be used for Full Text Searches performed with both + the Couchbase Web Console and the REST API. A detailed description of Query String-format is provided in + Query Types. +

      + +

      + For example, to search for instances of both nice and view, specify + "+nice +view" in a search from the Couchbase Web Console: +

      + +

      + +

      + +

      + When the search has returned, check in succession the show advanced query settings and show + command-line curl example checkboxes. The JSON for Query Request now displays the following: +

      + +

      + +

      + +

      + Copy the curl command displayed, by left-clicking on the Copy to Clipboard button. + Before attempting to execute the command from the command-line, paste it into a text-editor, and add + appropriate authentication-credentials. For example: +

      + + curl -u Administrator:password -XPOST -H "Content-Type: application/json" \ +http://localhost:8094/api/index/travel-sample-index/query \ +-d '{ + "explain": true, + "fields": [ + "*" + ], + "highlight": {}, + "query": { + "query": "{ \"+nice +view\" }" + } +}' + +

      + (For detailed information on Couchbase Server Role-Based Access Control, see + Authorization.) +

      + +

      + The code can now be again copied, pasted against the command-line, and executed; with the result-set appearing + as standard output. +

      + +

      + For additional assistance on Query String composition, left-click on the full text query syntax help link that + appears under the Search interactive text-field when show advanced query settings is checked: +

      + +

      + +

      + +

      + This link provides access to + page of + information on Query String Full Text Search queries. +

      + +
      + + + +
      + + + Further REST Examples + + +

      + Further examples of using the REST API to conduct Full Text Searches can be found in + Query Types. +

      + +
      + +
      + + + List of REST Features Supporting Full Text Search + + +

      + The full range of features for Full Text Search, as supported by the Couchbase REST API, is documented as + part of the REST API's reference information, on the page + Full Text Search API. +

      + +
      + + + +       diff --git a/content/fts/fts-sorting.dita b/content/fts/fts-sorting.dita index 88be6c93cc..c94635d06f 100644 --- a/content/fts/fts-sorting.dita +++ b/content/fts/fts-sorting.dita @@ -1,30 +1,110 @@ - Sorting Query Results + + + Sorting Query Results + + + + The results of Full Text Searches are returned as objects, the data within which can be sorted. + + -

      Starting Couchbase Server version 4.6, you can sort search results by any indexed field. In - previous releases, search results were always sorted by descending score so that highest - scoring results are listed first. This is still the default sort order, so if you don’t - specify a sort order, you’re unlikely to notice any difference.

      -

      You control sorting in your query JSON by specifying a sort field that - takes an array of either strings or objects. Strings designate the names fields to sort on, - e.g. "sort": ["fieldname"]. Objects are described below in

      -

      You can prefix any field name with the ‘-’ character, which causes that field to sorted in descending order. Items will first be sorted by the first field. Any items with the same value for that field, are then also sorted by the next field, and so on. All fields in the sort order MUST be indexed.

      -

      There are two special fields:

        -
      • _id - refers to the document identifier
        • -
      • _score - refers to the relevance score computed by Bleve
        • -

      -

      Here is a more complex example: - "sort": ["country", "state", "city","-_score"]

      -

      Results will first be sorted by country. If documents have the same value for country, they - will be sorted by state, and then, if country and state are the same, matches will be sorted - by city. Finally, if all other fields are equal, results will be sorted by score descending.

      -

      Here is a full query example that performs a query string query and demonstrates the proper - place to add the sort - property.{ + +

      + + + Sorting Result Data + + +

      + On query-completion, sorting allows specified members of the result-set to be displayed prior to others: this + facilitates review of the most significant data. +

      + +

      + Within a JSON query object, the required sort-type is specified by using the sort field. This + takes, as its value, an array of either strings or objects. +

      + +
      + +
      + + + Sorting with Strings + + +

      + The value of the sort field can be specified as an array of strings. These can be of three + types: +

      + +
        + +
      • + field name: Specifies the name of a field. If multiple fields are included in the array, the sorting + of documents begins according to their values for the field whose name is first in the array. If any number + of these values are identical, their documents are sorted again, this time according to their values for the + field whose name is second; then, if any number of these values are identical, their documents are sorted a third time, + this time according to their values for the field whose name is third; and so on. + +

        + Any document-field may be specified to hold the value on which sorting is to be based, provided that + the field has been indexed in some way, whether dynamically or specifically. +

        + +

        + The default sort-order is ascending. If a field-name is prefixed with the - character, + that field's results are sorted in descending order. +

        + +
        • + +
      • + _id: Refers to the document identifier. Whenever encountered in the array, causes sorting + to occur by document identifer. + +

        + +

        +
        • + +
      • + _score: Refers to the score assigned the document in the result-set. Whenever encountered in the + array, causes sorting to occur by score. + +

        + +

        + +
        • + +
      + +

      + For example: +

      + + "sort": ["country", "state", "city","-_score"] + +

      + This sort statement specifies that results will first be sorted by country. If + some documents are then found to have the same value in their country fields, they + are re-sorted by state. Next, if some of these documents are found to have the + same value in their + state fields, they are re-sorted + by city. Finally, if some of these documents are found to have the same value in their + city fields, they are re-sorted by score, in descending order. +

      + +

      + The following JSON query demonstrates how and where + the sort property can be specified: +

      + + { "explain": false, "fields": [ "title" @@ -34,34 +114,87 @@ "query":{ "query": "beautiful pool" } -}

      -
      Advanced Sort Options You can exercise - finer control over sort behavior using objects in the sort field.
        -
      • by Sort results on id, score, or a - field in the full text index.
        • -
      • field the name of the field to sort on, if you specify "by" : - "field". It is ignored if you specify score or - id.
        • -
      • missing specifies how you want search results sorted if they have no - value in a field used for sorting. Setting this sorts all results with missing values in - the sort field either before all other results (first) or after them - (last). Defaults to last.
        • +} + +
      + + +
      + + + Sorting with Objects + + +

      + Fine-grained control over sort-procedure can be achieved by specifying objects as array-values in + the sort field. Each object can have the following fields: +

      + +
        + +
      • + by: Sorts results on id, score, or a specified + field in the Full Text Index. + +

        + +

        + +
        • + +
      • + field: Specifies the name of a field on which to sort. Used only + if field has been specified as the value for the by + field; otherwise ignored. + +

        + +

        + +
        • +
      • - mode A single field in a full-text search index can contain multiple - values. This happens when you index an array with multiple elements or when a text - analyzer emits multiple tokens for an input. By default, such results are sorted in - default order, which is undefined but deterministic, allowing you to - page through results using from (offset) with reliable ordering. If you - need to control sort for multi-value fields, set mode to min or - max to sort using the minimum or maximum value.
        • -

      The example below demonstrates advanced sort settings. You can try it on the - travel-sample bucket if you create a default index that indexes all fields in hotel - documents. This query sorts search results based on - reviews.ratings.Overall, a field that is normally multi-valued because it + + missing: Specifies the sort-procedure for documents with a missing value + in a field specified for sorting. The value of missing can be first, in which case + results with missing values appear before other results; or last (the default), in which + case they appear after. + +

      + +

      + + + +
    • + mode: Specifies the search-order for index-fields that contain multiple values (in + consequence of + arrays or multi-token analyzer-output). The default order is undefined but + deterministic, allowing the paging of results from + from (offset), with + reliable ordering. To sort using the minimum or maximum value, the value of mode should be set + to either min or max. +
      • + +
    + +

    + The example below shows how an object-sort can be specified. It assumes that the + travel-sample bucket has been loaded, and a default index has been + created on it. This query sorts search-results based on + reviews.ratings.Overall — a field that is normally multi-valued, because it contains an array of different users' ratings. When there are multiple values, the highest - "Overall" ratings will be used for sorting. All hotels with no "Overall" ratings will be - sorted at the end.

    + Overall ratings are used for sorting. Hotels with no Overall rating are + placed at the end. +

    + +

    + For information on loading sample buckets, see + Install Sample Buckets. + For instructions on creating a default Full Text Index by means of the Couchbase Web Console, see + Performing Searches. +

    + { "explain": false, "fields": [ @@ -83,8 +216,13 @@ }, ] } -

    Finally, note that you can combine simple and advanced sort options freely in the - sort array.

    + +

    + Note also that + sort field accepts combinations of strings and objects as its value. This + is demonstrated as follows: +

    + { ... "sort": [ @@ -104,7 +242,9 @@ "-_score" ] } -

    +
    + +     diff --git a/content/fts/fts-text-analysis.dita b/content/fts/fts-text-analysis.dita deleted file mode 100644 index 5dd07738c5..0000000000 --- a/content/fts/fts-text-analysis.dita +++ /dev/null @@ -1,120 +0,0 @@ - - - - Text Analysis - -

    Text analysis is the process of transforming input text into a series of terms, - or canonical forms of words that are typically stripped of inflections and junk characters. - Analysis is done on the text documents when you create the index, and again on the text of - your query when you perform a search.

    -

    When people talk about full text search being "language aware", they are usually referring to the results of text analysis.

    -

    Example

    -

    Consider you have a document containing the text, "watery light beer" and you search for - "watered down".

    -

    There is no exact match in this case, but if you want this query to match this document you - can accomplish this by choosing the right analyzer. In this case, you want to use an analyzer - that can stem English words.

    -

    By doing this, the index entry for the document will contain the terms: "water", "light", "beer". And the query will search for terms: "water", "down".

    -

    Now you see that the terms will match and you get the expected search results.

    -

    Stemming is just one way in which the text can be transformed, the key here is that choosing the right analyzer is essential to getting high quality search results.

    -
    Analyzers -

    Analyzers are used to transform input text into a stream of tokens for indexing. Full Text - Search includes a number of analyzers or you can build your own. Analyzers are built up from - modular components.

      -
    • Character Filters strip out undesirable characters from the input. For example, the - "html" character filter ignores HTML tags and just indexes the content from HTML - documents.
      • -
    • Tokenizers split input strings into a token stream. Typically you’re trying to create - a token for each word.
      • -
    • Token Filters are chained together to perform additional post processing on the token - stream.
      • -

    -
    Tokenizers -

    Tokenizers cut sequences of characters into meaningful units (tokens) while ignoring useless characters, like spaces or commas.

    -

    Stemming is the process of reducing words to an underlying base form. The goal of stemming is to get rid of grammatical inflections while preserving the words’ meanings. Those underlying base forms are then written into the index.

    -

    In the Couchbase Server Web Console, you can find a list of available stemmers on the - Index > Full Text > - Edit or Create New Index > - Analyzers. If you click "New Analyzer" you - see a list of available stemmers in the "Token Filters" drop-down - list.

    -

    Single Token

    -

    The Single Token Tokenizer returns the entire input bytes as a single token. This can be - useful for things like URLs - (http://www.example.com/blog/stuff+and+nonsense/) or email address - (will.spammington@couchbase.com) that might otherwise be broken up at - punctuation or special character boundaries. It could also be used when multi-word phrases - should be indexed as a single term, for example, in the case of proper place names that - would otherwise be broken into multiple terms by the whitespace tokenizer.

    -

    Consider the excerpt below from the brewery documents in the beer-sample database. You can - use a single token analyzer for the city, name, - country, phone, or website - fields because they are all continuous strings that you may not want to divide at dots, - dashes, or spaces. The single token tokenizer is probably less useful for the name field, - which users will probably enter many different - ways.{ - "type": "brewery", - "name": "21st Amendment Brewery Cafe", - "city": "San Francisco", - "country": "United States", - "phone": "1-415-369-0900", - "website": "http://www.21st-amendment.com/", - ... - } -

    -

    Regular Expression

    -

    The Regular Expression Tokenizer will tokenize input using a configurable regular expression. The regular expression should match token text. See the Whitespace Tokenizer for an example of its usage. -

    -

    Whitespace

    -

    The Whitespace Tokenizer is an instance of the Regular Expression Tokenizer. It uses the - regular expression \w+. For many western languages this may work well as a - simple tokenizer.

    -

    Unicode

    -

    The Unicode Tokenizer uses the segment library to perform Unicode Text - Segmentation on word boundaries. It is recommended over the ICU tokenizer for all - languages not requiring dictionary-based tokenization that is supported by ICU.

    -

    Full Text Search includes a number of analyzers or you can build your own.

    -
    Customizing the Field Mappings -

    Dynamic Index Mapping

    -

    When the indexer encounters a field whose type you haven’t explicitly specified, it guesses - the type by looking at the JSON value. - - - - - - Type of JSON value - Indexed as... - - - - - Boolean - Boolean - - - Number - Number - - - String containing a date - Date - - - String (not containing a date) - String - - - -
    The indexer attempts to parse String values as dates and indexes them as such if - the operation succeeds. Full text search and Bleve expect dates to be in the format - specified by RFC-3339, which is a specific profile of ISO-8601 that is more - restrictive.

    -

    Note that other String values like "7" or "true" will never be indexed as a number or - Boolean. The number type is modeled as 64-bit floating point value internally.

    - -     diff --git a/content/fts/fts-using-analyzers.dita b/content/fts/fts-using-analyzers.dita new file mode 100644 index 0000000000..8fa1fa31d2 --- /dev/null +++ b/content/fts/fts-using-analyzers.dita @@ -0,0 +1,723 @@ + + + + + + Understanding Analyzers + + + + Analyzers increase search-awareness by transforming input text into token-streams, which + permit the management of richer and more finely controlled forms of text-matching. An + analyzer consists of modules, each of + which performs a particular, sequenced role in the transformation. + + + + +
    + + Principles of Text-Analysis + + +

    + Analyzers pre-process input-text submitted for Full Text Search; typically, by removing + characters that might prohibit certain match-options. Analysis is performed on document-contents + when indexes are created; and is also performed on the input-text submitted for a search. The + benefit of analysis is often referred to as language awareness. +

    + +

    + For example, if the input-text for a search is enjoyed staying here, and the document-content + contains the phrase many enjoyable activities, the dictionary-based words do not permit + a match. However, by using an analyzer that (by means of its inner Token Filter component) + stems words, the input-text yields the tokens + enjoy, stay, and here; while the document-content yields the tokens + enjoy and activ. By means of the common token enjoy, this + permits a match + between enjoyed and enjoyable. +

    + +

    + Since different analyzers pre-process text in different ways, effective Full Text Search depends on the right choice + of analyzer, for the type of matches that are desired. +

    + +

    + Couchbase Full Text Search provides a number of pre-constructed analyzers that can be used with + Full Text Indexes. Additionally, analyzers can be custom-created, by means of the Couchbase Web Console. The + remainder of this page explains the architecture of analyzers, and describes the modular components that + Couchbase Full Text Search makes available for custom-creation. It also lists the pre-constructed analyzers + that are available, and describes the modules that they contain. +

    + +

    + For examples of both selecting and custom-creating analyzers by means of the Couchbase Web Console, see + Creating Indexes. +

    + +
    + +
    + + + Analyzer Architecture + + +

    + Analyzers are built from modular components: +

    + +
      + +
    • + Character Filters + remove undesirable characters from input: for example, the html + character filter removes HTML tags, and indexes HTML text-content alone. +
      • + +
    • + Tokenizers + split input-strings into individual + tokens, which together are made into a token stream. The nature of the decision-making whereby splits are + made differs across tokenizers. +
      • + +
    • + Token Filters are chained together, with each performing additional post-processing on each token + in the stream provided by the tokenizer. This may include reducing tokens to the stems of the + dictionary-based words from which they were derived, removing any remaining punctuation from tokens, and removing + certain tokens deemed unnecessary. +
      • + +
    + +

    + Each component-type is described in more detail below. Note that these components can be used to custom-create an analyzer + by means of the Couchbase Web Console. This is explained and exemplified in + Creating Indexes. +

    + +
    + +
    + + + Character Filters + + +

    + Character Filters remove undesirable characters. The following filters are available: +

    + +
      + +
    • + html: Removes html elements such as <p>, and decodes expressions such as + &amp; to their appropriate text equivalent. + +

      + +

      + +
      • + +
    • + zero_width_spaces: Substitutes a regular space-character for each zero-width non-joiner space. + +

      + +

      + +
      • + +
    + +
    + +
    + + + Tokenizers + + +

    + Tokenizers split input-strings into individual tokens: characters likely to prohibit certain kinds + of matching (for example, spaces or commas) are omitted. The tokens so created are then made into + a token stream for the query. +

    + +

    + The following tokenizers are available from the Couchbase Web Console: +

    + +
      +
    • + Letter: Creates tokens by breaking input-text into subsets that consist of letters only: + characters such as punctuation-marks and numbers are omitted. Creation of a token ends whenever + a non-letter character is encountered. For example, the text Reqmnt: 7-element phrase + would return the following tokens: Reqmnt, element, and + phrase. + +

      + +

      + +
      • + +
    • + Single: Creates a single token from the entirety of the input-text. For example, the text + in each place would return the following token: in each place. + Note that this may be useful for handling URLs or email-addresses, which can thus be prevented + from being broken at punctuation or special-character boundaries. It may also be used to prevent + multi-word phrases (for example, placenames such as Milton Keynes or + San Francisco) from being broken up due to whitespace; so that they become indexed + as a single term. + +

      + +

      + +
      • + +
    • + Unicode: Creates tokens by performing Unicode Text Segmentation on word-boundaries, using the + segment library. + For examples, see + Unicode Word Boundaries. + +

      + +

      + +
      • + +
    • + Web: Creates tokens by identifying and removing html tags. For example, the text + <h1>Introduction<\h1> would return the token Introduction. + +

      + +

      + +
      • + +
    • + Whitespace: Creates tokens by breaking input-text into subsets according to where + whitespace occurs. For example, the text in each place would return the following + tokens: in, each, and place. + +

      + +

      + +
      • + +
    + +
    + +
    + + + Token Filters + + +

    + Token Filters accept a token-stream provided by a tokenizer, and make modifications to the tokens in the stream. +

    + +

    + A frequently used form of token filtering is stemming; this reduces words + to a base form that typically consists of the initial stem of the word (for example, + play, which is the stem of player, playing, + playable, and more). With the stem used as the token, a wider variety of matches + can be made (for example, the input-text player can be matched with the document-content + playable). +

    + +

    + The following kinds of token-filtering are supported by Couchbase Full Text Search: +

    + +
      + +
    • + dict_compound: Allows user-specification of a dictionary whose words can be combined + into compound forms, and individually indexed. + +

      + +

      + +
      • + +
    • + apostrophe: Removes all characters after an apostrophe, and the apostrophe itself. For example, + they've becomes they. + +

      + +

      +
      • + +
    • + elision: Identifies and removes characters that prefix a term and are separated + from it by an apostrophe. For example, + in French, l'avion becomes avion. + +

      + +

      +
      • + +
    • + edge_ngram: From each token, computes + n-grams that + are rooted either at the front or the back. + +

      + +

      +
      • + +
    • + keyword_marker: Identifies keywords and marks them as such. These are then ignored by any + downstream stemmer. + +

      + +

      +
      • + +
    • + normalize_unicode: Converts tokens into + Unicode Normalization Form. + +

      + +

      +
      • + +
    • + ngram: From each token, computes + n-grams. + There are two parameters, which are the minimum and maximum n-gram length. + +

      + +

      +
      • + +
    • + shingle: Computes multi-token shingles from the token stream. For example, the token stream + the quick brown fox, when configured with a shingle minimum and a shingle + maximum length of 2, + produces the tokens the quick, quick brown, and brown fox. + +

      + +

      +
      • + + +
    • + stemmer: Uses + libstemmer to reduce + tokens to word-stems. + +

      + +

      +
      • + +
    • + stop_tokens: Removes from the stream tokens considered unnecessary for a + Full Text Search: for example, and, is, and the. + +

      + +

      +
      • + +
    • + to_lower: Converts all characters to lower case. For example, HTML becomes html. + +

      + +

      +
      • + +
    • + truncate: Truncates each token to a maximum-permissible token-length. + +

      + +

      +
      • + +
    • + possessive: Removes English possessives. For example, software's + becomes software. + +

      + +

      +
      • + +
    + +

    + Note that token filters are frequently configured according to the special characteristics of individual languages. + Couchbase Full Text Search provides multiple language-specific versions of the elision, normalize, + stemmer, possessive, and stop token filters. Specially supported languages are shown in + the table immediately below. +

    + + + + Token-Filter Lanaguages Supported by Couchbase Server 5.5 + + + + + + + + Name + Language + + + + + + + ar + Arabic + + + + bg + Bulgarian + + + + ca + Catalan + + + + cjk + Chinese | Japanese | Korean + + + + ckb + Kurdish + + + + da + Danish + + + + de + German + + + + el + Greek + + + + en + English + + + + es + Spanish (Castilian) + + + + eu + Basque + + + + fa + Persian + + + + fi + Finnish + + + + fr + French + + + + ga + Gaelic + + + + gl + Spanish (Galician) + + + + hi + Hindi + + + + hu + Hungarian + + + + hy + Armenian + + + + id, in + Indonesian + + + + it + Italian + + + + nl + Dutch + + + + no + Norwegian + + + + pt + Portuguese + + + + ro + Romanian + + + + ru + Russian + + + + sv + Swedish + + + + tr + Turkish + + + + +
    + +
    + +
    + + + Pre-Constructed Analyzers + + +

    + A number of pre-constructed analyzers are available, and can be selected from the Couchbase Web Console. For examples + of selection, see + Creating Indexes. The basic analyzers are + as follows. See the sections above for details on the referenced analyzer-components. +

    + +
      +
    • + keyword: Creates a single token representing the entire input, which is otherwise + unchanged. This forces exact matches, and preserves characters such as spaces. + +

      + +

      + +
      • + +
    • + simple: Analysis by means of the Unicode tokenizer and the to_lower token filter. +

      + +

      + +
      • + +
    • + standard: Analysis by means of the Unicode tokenizer, the to_lower token filter, and the stop token filter. + +

      + +

      + +
      • + + +
    • + web: Analysis by means of the Web tokenizer and the to_lower token filter. + +

      + +

      + +
      • + +
    + +

    + Additionally, a range of analyzers is provided for the specific support of certain languages; as shown by + the table immediately below. +

    + + + + Analyzer-Lanaguages Supported by Couchbase Server 5.5 + + + + + + + + Name + Language + + + + + + + ar + Arabic + + + + cjk + Chinese | Japanese | Korean + + + + ckb + Kurdish + + + + da + Danish + + + + de + German + + + + en + English + + + + es + Spanish (Castilian) + + + + fa + Persian + + + + fi + Finnish + + + + fr + French + + + + hi + Hindi + + + + hu + Hungarian + + + + it + Italian + + + + nl + Dutch + + + + no + Norwegian + + + + pt + Portuguese + + + + ro + Romanian + + + + ru + Russian + + + + sv + Swedish + + + + tr + Turkish + + + + +
    + +
    + + + +     diff --git a/content/fts/full-text-intro.dita b/content/fts/full-text-intro.dita index 9860471deb..4e690e9e39 100644 --- a/content/fts/full-text-intro.dita +++ b/content/fts/full-text-intro.dita @@ -1,88 +1,133 @@ - Full Text Search Reference + + + Full Text Search: Fundamentals + + + + Full Text Search (FTS) lets you create, manage, and query specially purposed indexes, + defined on JSON documents within a Couchbase bucket. + + -

    Couchbase Server Full Text Search (FTS) enables you to create, manage, and query full text - indexes on JSON documents stored in a Couchbase bucket.

    -

    Full Text Search provides powerful capabilities for querying natural language. There are - three major benefits that make full text search much easier to use than alternative methods of - querying, such as a wildcard match in N1QL using LIKE statements:

      -
    • FTS provides a language-aware manner of searching, so users can type - "beauties" and find results for "beauty" - which is - not easy to do with regular expressions.
      • -
    • FTS provides scoring (relevance ranking) for results. Text search often returns massive - numbers of matching documents, which are ordered by score so that users are presented with - only a small number of results that are most likely to be the ones they are looking for. - With regular expressions, there’s no practical way to simulate relevance.
      • -
    • FTS provides fast indexes that support a wide range of possible text searches. Regular - expressions can be used to build one-off indexes but can’t be used to build indexes that - support a full range of possible text searches.
      • -

    -

    Couchbase Server Full Text Search is powered by Bleve, a full text - search and indexing library written in Go. FTS indexes your documents with Bleve and lets you use Bleve’s many powerful query types:

      -
    • Term, Phrase, Match, Match Phrase, Prefix
      • -
    • Conjunction, Disjunction, Boolean
      • -
    • Numeric and Date Ranges
      • -
    • Query String (see Syntax)
      • -
    Analyzers are a key component of search indexing (see ). FTS includes Bleve’s general-purpose analyzers as well as pre-built text analyzers for the following languages: Arabic, English, French, German, Hindi, Italian, Persian, Portuguese, Sorani, Spanish.

    -

    FTS scores documents for relevance using Bleve’s industry standard tf-idf scoring with query time boosting.

    -
    Executing Your First Full Text Query -

    The Full Text Search service needs to be enabled when adding a node to a cluster during configuration. - New nodes have the service enabled by default, just like data, index and N1QL query services. - On Production systems we highly recommend limiting the FTS service to an independent set of nodes to - help manage resources and performance.

    -

    Creating Your First Full Text Index

    -
    1. Log in to the Couchbase Web Console, for example, http://localhost:8091/. -

      Click the sidebar tab - Search - , then click on the New Full Text Index - button.

      A form where you can define your new index configuration is - displayed.

      2. -
    2. Each index needs a unique name. Type one in, such as "test-index". - Only alphanumeric characters, hyphens, and underscores are allowed for index names. Also, the - first character must be an alphabetic character.
      3. -
    3. Choose a bucket from the drop-down list.

      There are more options, but you can ignore those - for now and rely on the default values.

      4. -
    4. Go! Finally, click the Create Index button.

      You should now see - a summary page of your new full text index. The document count shows how many - documents have been indexed so far.

      5. -
    -

    Querying Your Full Text Index

    -
      -
    1. Click on the Search tab again.
      2. -
    2. From the list, select your newly created index.
      3. -
    3. In the query field, type in a query term and press the Search - button.
    -

    Congratulations, you’ve just executed your first full text query!

    + +
    + + + Features of Full Text Search + + +

    + Full Text Search provides extensive capabilities for natural-language querying. + These include: +

    + +
      + +
    • + Language-aware searching; allowing users to search for, say, the word + beauties, and additionally obtain results + for beauty and beautiful. + +

      + +

      +
      • + +
    • + Scoring of results, according to relevancy; allowing + users to obtain result-sets that only contain documents awarded the highest + scores. This keeps result-sets manageably small, even when the total number of documents + returned is extremely large. + +

      + +

      +
      • + +
    • + Fast indexes, which support a wide range of possible text-searches. + +

      + +

      +
      • + + + +
    + +

    + Full Text Search is powered by Bleve, an open source + search and indexing library written in Go. Full Text Search uses + Bleve for the indexing of documents, + and also makes available Bleve’s extensive range of query types. These include: +

    + +
      +
    • + Match, Match Phrase, Doc ID, and Prefix queries +
      • + +
    • + Conjunction, Disjunction, and Boolean field queries +
      • + +
    • + Numeric Range and Date Range queries +
      • + +
    • + Geospatial queries +
      • + +
    • + Query String queries, which employ a special syntax to express the details of each query (see + Query String Query + for information) +
      • + +
    + +

    + Full Text Search includes pre-built text analyzers for the + following languages: Arabic, CJK characters (Chinese, Japanese, and Korean), English, French, Hindi, Italian, Kurdish, Persian, and Portuguese. + Additional languages have been added to Couchbase Server 5.5. +

    +
    -
    Using the Full Text Search REST API -

    You can also use the full text search REST APIs to access your index. For example, if your - index is named myFirstIndex, here's how you can use curl to check how - many documents are indexed: - curl -u Administrator:password http://localhost:8094/api/index/myFirstIndex/count - Here's an example of using curl to query the index myFirstIndex: - curl -X POST -H "Content-Type: application/json" \ - -d '{"size":10,"query":{"query":"your search string"}}' \ - http://localhost:8094/api/index/myFirstIndex/query - That's it! You should be ready to explore, or read on to learn more about creating and - querying indexes.

    -
    - Also, you may want to take a look at the following blog that explains how the full text search API (experimental) is expressed using the Java SDK: - - - + +
    + + + Recent Feature-Additions + + +

    + The following features have recently been added: +

    + +
    + +
    + + + Authorization for Full Text Search + + +

    + To access Full Text Search, users require appropriate roles. + The role FTS Admin + must therefore be assigned to those who intend to create indexes; and the role FTS Searcher to those who intend to + perform searches. For information on creating users and assigning roles, see + Authorization. +

    + +
    + + + diff --git a/content/fts/full-text-search.ditamap b/content/fts/full-text-search.ditamap index b84de0cc37..650e68501b 100644 --- a/content/fts/full-text-search.ditamap +++ b/content/fts/full-text-search.ditamap @@ -1,16 +1,31 @@ + Full Text Search - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/fts/images/ft-add-type-mapping.png b/content/fts/images/ft-add-type-mapping.png new file mode 100644 index 0000000000..eae69e863f Binary files /dev/null and b/content/fts/images/ft-add-type-mapping.png differ diff --git a/content/fts/images/ft-type-mappings-ui-closed.png b/content/fts/images/ft-type-mappings-ui-closed.png new file mode 100644 index 0000000000..e1c3bdb97e Binary files /dev/null and b/content/fts/images/ft-type-mappings-ui-closed.png differ diff --git a/content/fts/images/fts-add-alias-screen.png b/content/fts/images/fts-add-alias-screen.png new file mode 100644 index 0000000000..79500bae06 Binary files /dev/null and b/content/fts/images/fts-add-alias-screen.png differ diff --git a/content/fts/images/fts-add-index-button.png b/content/fts/images/fts-add-index-button.png new file mode 100644 index 0000000000..2dcb1f0fb8 Binary files /dev/null and b/content/fts/images/fts-add-index-button.png differ diff --git a/content/fts/images/fts-add-index-initial.png b/content/fts/images/fts-add-index-initial.png new file mode 100644 index 0000000000..6a4523e323 Binary files /dev/null and b/content/fts/images/fts-add-index-initial.png differ diff --git a/content/fts/images/fts-add-tokenizer-pulldown.png b/content/fts/images/fts-add-tokenizer-pulldown.png new file mode 100644 index 0000000000..d1cd0307ad Binary files /dev/null and b/content/fts/images/fts-add-tokenizer-pulldown.png differ diff --git a/content/fts/images/fts-advanced-panel.png b/content/fts/images/fts-advanced-panel.png new file mode 100644 index 0000000000..5c88e385a0 Binary files /dev/null and b/content/fts/images/fts-advanced-panel.png differ diff --git a/content/fts/images/fts-advanced-query-settings.png b/content/fts/images/fts-advanced-query-settings.png new file mode 100644 index 0000000000..d318672450 Binary files /dev/null and b/content/fts/images/fts-advanced-query-settings.png differ diff --git a/content/fts/images/fts-analyzers-panel-initial.png b/content/fts/images/fts-analyzers-panel-initial.png new file mode 100644 index 0000000000..fe8edbe687 Binary files /dev/null and b/content/fts/images/fts-analyzers-panel-initial.png differ diff --git a/content/fts/images/fts-analyzers-panel-select-char-filter.png b/content/fts/images/fts-analyzers-panel-select-char-filter.png new file mode 100644 index 0000000000..5b0d7b044a Binary files /dev/null and b/content/fts/images/fts-analyzers-panel-select-char-filter.png differ diff --git a/content/fts/images/fts-analyzers-panel-select-token-filter.png b/content/fts/images/fts-analyzers-panel-select-token-filter.png new file mode 100644 index 0000000000..1ea2a412f3 Binary files /dev/null and b/content/fts/images/fts-analyzers-panel-select-token-filter.png differ diff --git a/content/fts/images/fts-analyzers-panel-subsequent.png b/content/fts/images/fts-analyzers-panel-subsequent.png new file mode 100644 index 0000000000..babccacd4c Binary files /dev/null and b/content/fts/images/fts-analyzers-panel-subsequent.png differ diff --git a/content/fts/images/fts-console-initial.png b/content/fts/images/fts-console-initial.png new file mode 100644 index 0000000000..362a3b0a8c Binary files /dev/null and b/content/fts/images/fts-console-initial.png differ diff --git a/content/fts/images/fts-create-index-button.png b/content/fts/images/fts-create-index-button.png new file mode 100644 index 0000000000..744cc4062e Binary files /dev/null and b/content/fts/images/fts-create-index-button.png differ diff --git a/content/fts/images/fts-custom-analyzer-dialog-initial.png b/content/fts/images/fts-custom-analyzer-dialog-initial.png new file mode 100644 index 0000000000..a75be3b714 Binary files /dev/null and b/content/fts/images/fts-custom-analyzer-dialog-initial.png differ diff --git a/content/fts/images/fts-custom-character-filter-dialog-filled.png b/content/fts/images/fts-custom-character-filter-dialog-filled.png new file mode 100644 index 0000000000..114164e397 Binary files /dev/null and b/content/fts/images/fts-custom-character-filter-dialog-filled.png differ diff --git a/content/fts/images/fts-custom-character-filter-dialog-initial.png b/content/fts/images/fts-custom-character-filter-dialog-initial.png new file mode 100644 index 0000000000..47157f85e7 Binary files /dev/null and b/content/fts/images/fts-custom-character-filter-dialog-initial.png differ diff --git a/content/fts/images/fts-custom-date-time-parser-dialog.png b/content/fts/images/fts-custom-date-time-parser-dialog.png new file mode 100644 index 0000000000..179b11c647 Binary files /dev/null and b/content/fts/images/fts-custom-date-time-parser-dialog.png differ diff --git a/content/fts/images/fts-custom-filters-panel-initial.png b/content/fts/images/fts-custom-filters-panel-initial.png new file mode 100644 index 0000000000..c5e619cb8b Binary files /dev/null and b/content/fts/images/fts-custom-filters-panel-initial.png differ diff --git a/content/fts/images/fts-custom-filters-panel-new-character-filter.png b/content/fts/images/fts-custom-filters-panel-new-character-filter.png new file mode 100644 index 0000000000..b62ab9705b Binary files /dev/null and b/content/fts/images/fts-custom-filters-panel-new-character-filter.png differ diff --git a/content/fts/images/fts-custom-filters-panel-new-token-filter.png b/content/fts/images/fts-custom-filters-panel-new-token-filter.png new file mode 100644 index 0000000000..4aff577ae3 Binary files /dev/null and b/content/fts/images/fts-custom-filters-panel-new-token-filter.png differ diff --git a/content/fts/images/fts-custom-filters-panel-new-tokenizer.png b/content/fts/images/fts-custom-filters-panel-new-tokenizer.png new file mode 100644 index 0000000000..d3a9232d13 Binary files /dev/null and b/content/fts/images/fts-custom-filters-panel-new-tokenizer.png differ diff --git a/content/fts/images/fts-custom-filters-panel-new-word-list.png b/content/fts/images/fts-custom-filters-panel-new-word-list.png new file mode 100644 index 0000000000..d4081e09c1 Binary files /dev/null and b/content/fts/images/fts-custom-filters-panel-new-word-list.png differ diff --git a/content/fts/images/fts-custom-filters-token-filter-dialog-complete.png b/content/fts/images/fts-custom-filters-token-filter-dialog-complete.png new file mode 100644 index 0000000000..37d279ff8f Binary files /dev/null and b/content/fts/images/fts-custom-filters-token-filter-dialog-complete.png differ diff --git a/content/fts/images/fts-custom-filters-token-filter-dialog-initial.png b/content/fts/images/fts-custom-filters-token-filter-dialog-initial.png new file mode 100644 index 0000000000..6f0b23b289 Binary files /dev/null and b/content/fts/images/fts-custom-filters-token-filter-dialog-initial.png differ diff --git a/content/fts/images/fts-custom-filters-token-filter-types.png b/content/fts/images/fts-custom-filters-token-filter-types.png new file mode 100644 index 0000000000..b626075a6d Binary files /dev/null and b/content/fts/images/fts-custom-filters-token-filter-types.png differ diff --git a/content/fts/images/fts-custom-filters-tokenizer-dialog-completed.png b/content/fts/images/fts-custom-filters-tokenizer-dialog-completed.png new file mode 100644 index 0000000000..922f2d41a8 Binary files /dev/null and b/content/fts/images/fts-custom-filters-tokenizer-dialog-completed.png differ diff --git a/content/fts/images/fts-custom-filters-tokenizer-dialog-initial.png b/content/fts/images/fts-custom-filters-tokenizer-dialog-initial.png new file mode 100644 index 0000000000..501e77f19d Binary files /dev/null and b/content/fts/images/fts-custom-filters-tokenizer-dialog-initial.png differ diff --git a/content/fts/images/fts-custom-wordlist-dialog-complete.png b/content/fts/images/fts-custom-wordlist-dialog-complete.png new file mode 100644 index 0000000000..eeee2c165b Binary files /dev/null and b/content/fts/images/fts-custom-wordlist-dialog-complete.png differ diff --git a/content/fts/images/fts-custom-wordlist-dialog-initial.png b/content/fts/images/fts-custom-wordlist-dialog-initial.png new file mode 100644 index 0000000000..44fce2f7a1 Binary files /dev/null and b/content/fts/images/fts-custom-wordlist-dialog-initial.png differ diff --git a/content/fts/images/fts-date-time-parser-initial.png b/content/fts/images/fts-date-time-parser-initial.png new file mode 100644 index 0000000000..e7e7eee3d3 Binary files /dev/null and b/content/fts/images/fts-date-time-parser-initial.png differ diff --git a/content/fts/images/fts-fts-console-initial.png b/content/fts/images/fts-fts-console-initial.png new file mode 100644 index 0000000000..22cec52abc Binary files /dev/null and b/content/fts/images/fts-fts-console-initial.png differ diff --git a/content/fts/images/fts-full-text-aliases-page-with-alias.png b/content/fts/images/fts-full-text-aliases-page-with-alias.png new file mode 100644 index 0000000000..0f5a40a584 Binary files /dev/null and b/content/fts/images/fts-full-text-aliases-page-with-alias.png differ diff --git a/content/fts/images/fts-full-text-aliases-panel.png b/content/fts/images/fts-full-text-aliases-panel.png new file mode 100644 index 0000000000..ca34635f3f Binary files /dev/null and b/content/fts/images/fts-full-text-aliases-panel.png differ diff --git a/content/fts/images/fts-geopoint-definition.png b/content/fts/images/fts-geopoint-definition.png new file mode 100644 index 0000000000..ee442d3c9f Binary files /dev/null and b/content/fts/images/fts-geopoint-definition.png differ diff --git a/content/fts/images/fts-index-definition-preview.png b/content/fts/images/fts-index-definition-preview.png new file mode 100644 index 0000000000..23db6b2039 Binary files /dev/null and b/content/fts/images/fts-index-definition-preview.png differ diff --git a/content/fts/images/fts-index-management-ui.png b/content/fts/images/fts-index-management-ui.png new file mode 100644 index 0000000000..66fdc00e41 Binary files /dev/null and b/content/fts/images/fts-index-management-ui.png differ diff --git a/content/fts/images/fts-index-name-and-bucket.png b/content/fts/images/fts-index-name-and-bucket.png new file mode 100644 index 0000000000..df50b41f54 Binary files /dev/null and b/content/fts/images/fts-index-name-and-bucket.png differ diff --git a/content/fts/images/fts-index-replicas-error-message.png b/content/fts/images/fts-index-replicas-error-message.png new file mode 100644 index 0000000000..312a378129 Binary files /dev/null and b/content/fts/images/fts-index-replicas-error-message.png differ diff --git a/content/fts/images/fts-index-replicas-interface.png b/content/fts/images/fts-index-replicas-interface.png new file mode 100644 index 0000000000..1ef9b6bc59 Binary files /dev/null and b/content/fts/images/fts-index-replicas-interface.png differ diff --git a/content/fts/images/fts-index-replicas-ui.png b/content/fts/images/fts-index-replicas-ui.png new file mode 100644 index 0000000000..fb540ddfa3 Binary files /dev/null and b/content/fts/images/fts-index-replicas-ui.png differ diff --git a/content/fts/images/fts-index-type-interface.png b/content/fts/images/fts-index-type-interface.png new file mode 100644 index 0000000000..3cfa47e3df Binary files /dev/null and b/content/fts/images/fts-index-type-interface.png differ diff --git a/content/fts/images/fts-login-screen.png b/content/fts/images/fts-login-screen.png new file mode 100644 index 0000000000..5984b3edbc Binary files /dev/null and b/content/fts/images/fts-login-screen.png differ diff --git a/content/fts/images/fts-new-index-progress.png b/content/fts/images/fts-new-index-progress.png new file mode 100644 index 0000000000..a41db9532a Binary files /dev/null and b/content/fts/images/fts-new-index-progress.png differ diff --git a/content/fts/images/fts-query-string-query-at-ui.png b/content/fts/images/fts-query-string-query-at-ui.png new file mode 100644 index 0000000000..986571c46b Binary files /dev/null and b/content/fts/images/fts-query-string-query-at-ui.png differ diff --git a/content/fts/images/fts-query-string-results-at-ui.png b/content/fts/images/fts-query-string-results-at-ui.png new file mode 100644 index 0000000000..cbd6d59364 Binary files /dev/null and b/content/fts/images/fts-query-string-results-at-ui.png differ diff --git a/content/fts/images/fts-query-syntax-help-link.png b/content/fts/images/fts-query-syntax-help-link.png new file mode 100644 index 0000000000..f6d93ebce5 Binary files /dev/null and b/content/fts/images/fts-query-syntax-help-link.png differ diff --git a/content/fts/images/fts-select-search-tab.png b/content/fts/images/fts-select-search-tab.png new file mode 100644 index 0000000000..dbabfabe36 Binary files /dev/null and b/content/fts/images/fts-select-search-tab.png differ diff --git a/content/fts/images/fts-type-identifier-ui.png b/content/fts/images/fts-type-identifier-ui.png new file mode 100644 index 0000000000..cbc2a8287b Binary files /dev/null and b/content/fts/images/fts-type-identifier-ui.png differ diff --git a/content/fts/images/fts-type-mappings-child-field-dialog-complete.png b/content/fts/images/fts-type-mappings-child-field-dialog-complete.png new file mode 100644 index 0000000000..ac5c25dcf7 Binary files /dev/null and b/content/fts/images/fts-type-mappings-child-field-dialog-complete.png differ diff --git a/content/fts/images/fts-type-mappings-child-field-dialog.png b/content/fts/images/fts-type-mappings-child-field-dialog.png new file mode 100644 index 0000000000..37b6ea7d95 Binary files /dev/null and b/content/fts/images/fts-type-mappings-child-field-dialog.png differ diff --git a/content/fts/images/fts-type-mappings-child-field-saved.png b/content/fts/images/fts-type-mappings-child-field-saved.png new file mode 100644 index 0000000000..54fab4c173 Binary files /dev/null and b/content/fts/images/fts-type-mappings-child-field-saved.png differ diff --git a/content/fts/images/fts-type-mappings-child-mapping-add-field.png b/content/fts/images/fts-type-mappings-child-mapping-add-field.png new file mode 100644 index 0000000000..7280ed8262 Binary files /dev/null and b/content/fts/images/fts-type-mappings-child-mapping-add-field.png differ diff --git a/content/fts/images/fts-type-mappings-child-mapping-dialog-complete.png b/content/fts/images/fts-type-mappings-child-mapping-dialog-complete.png new file mode 100644 index 0000000000..e568c78844 Binary files /dev/null and b/content/fts/images/fts-type-mappings-child-mapping-dialog-complete.png differ diff --git a/content/fts/images/fts-type-mappings-child-mapping-dialog.png b/content/fts/images/fts-type-mappings-child-mapping-dialog.png new file mode 100644 index 0000000000..94712b49bc Binary files /dev/null and b/content/fts/images/fts-type-mappings-child-mapping-dialog.png differ diff --git a/content/fts/images/fts-type-mappings-ui-add.png b/content/fts/images/fts-type-mappings-ui-add.png new file mode 100644 index 0000000000..73d7cc2a66 Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-add.png differ diff --git a/content/fts/images/fts-type-mappings-ui-addition-both-checked.png b/content/fts/images/fts-type-mappings-ui-addition-both-checked.png new file mode 100644 index 0000000000..83cfbbf09c Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-addition-both-checked.png differ diff --git a/content/fts/images/fts-type-mappings-ui-addition-both-unchecked.png b/content/fts/images/fts-type-mappings-ui-addition-both-unchecked.png new file mode 100644 index 0000000000..ef618a287a Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-addition-both-unchecked.png differ diff --git a/content/fts/images/fts-type-mappings-ui-addition-default-unchecked.png b/content/fts/images/fts-type-mappings-ui-addition-default-unchecked.png new file mode 100644 index 0000000000..72aa358bc9 Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-addition-default-unchecked.png differ diff --git a/content/fts/images/fts-type-mappings-ui-analyzers-menu.png b/content/fts/images/fts-type-mappings-ui-analyzers-menu.png new file mode 100644 index 0000000000..e8dd91ecf0 Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-analyzers-menu.png differ diff --git a/content/fts/images/fts-type-mappings-ui-closed.png b/content/fts/images/fts-type-mappings-ui-closed.png new file mode 100644 index 0000000000..36618876c6 Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-closed.png differ diff --git a/content/fts/images/fts-type-mappings-ui-edit.png b/content/fts/images/fts-type-mappings-ui-edit.png new file mode 100644 index 0000000000..2ca7515c5c Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-edit.png differ diff --git a/content/fts/images/fts-type-mappings-ui-field-options.png b/content/fts/images/fts-type-mappings-ui-field-options.png new file mode 100644 index 0000000000..374a2610db Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-field-options.png differ diff --git a/content/fts/images/fts-type-mappings-ui-fields-buttons.png b/content/fts/images/fts-type-mappings-ui-fields-buttons.png new file mode 100644 index 0000000000..186c7dc23d Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-fields-buttons.png differ diff --git a/content/fts/images/fts-type-mappings-ui-select-data-type.png b/content/fts/images/fts-type-mappings-ui-select-data-type.png new file mode 100644 index 0000000000..29ef13d15b Binary files /dev/null and b/content/fts/images/fts-type-mappings-ui-select-data-type.png differ diff --git a/content/fts/images/fts-ui-curl-example.png b/content/fts/images/fts-ui-curl-example.png new file mode 100644 index 0000000000..3e29b75202 Binary files /dev/null and b/content/fts/images/fts-ui-curl-example.png differ diff --git a/content/fts/images/fts-ui-query-scores-display.png b/content/fts/images/fts-ui-query-scores-display.png new file mode 100644 index 0000000000..4dcdaf55c5 Binary files /dev/null and b/content/fts/images/fts-ui-query-scores-display.png differ diff --git a/content/fts/images/fts-ui-search-for-term.png b/content/fts/images/fts-ui-search-for-term.png new file mode 100644 index 0000000000..2d270c979b Binary files /dev/null and b/content/fts/images/fts-ui-search-for-term.png differ diff --git a/content/fts/images/fts-ui-search-results-page.png b/content/fts/images/fts-ui-search-results-page.png new file mode 100644 index 0000000000..b5c998710e Binary files /dev/null and b/content/fts/images/fts-ui-search-results-page.png differ diff --git a/content/functions/overview.adoc b/content/functions/overview.adoc new file mode 100644 index 0000000000..e69de29bb2 diff --git a/content/functions/template.adoc b/content/functions/template.adoc new file mode 100644 index 0000000000..1d4e10f680 --- /dev/null +++ b/content/functions/template.adoc @@ -0,0 +1,20 @@ +[#page-title] += PageTitle + +[abstract] +Short description for the page. + +Paragragh goes here. + +Use this format if you need a list: +* One +* Two + +== Section Heading + +Content for the section. + +== Another section + +Content for the section. + diff --git a/content/getting-started/choose-your-next-steps.dita b/content/getting-started/choose-your-next-steps.dita index 5dd2be5a6e..bd32604376 100644 --- a/content/getting-started/choose-your-next-steps.dita +++ b/content/getting-started/choose-your-next-steps.dita @@ -5,185 +5,123 @@ - - Choose Your Next Steps - + Choose Your Next Steps - - Now, to complete the Getting Started sequence: consider your options as to - what to do next, in order to continue improving your knowledge. The Couchbase documentation-set - provides detailed information on all aspects - of the system; and this section lists some of the important areas you can visit. - + To complete the Getting Started sequence, consider your options as to what to do + next to continue improving your knowledge. The Couchbase documentation set provides detailed + information on all aspects of the platform; and this topic lists some of the important areas + you can visit.
    - - Where to Learn More - + Where to Learn More -

    - By navigating to the sections listed below, you can increase your knowledge of the areas - you've already looked at — - installation, configuration, and N1QL-querying — and also - learn about new and (in some cases) quite advanced topics; such as multi-node clustering, fail-over, replication, and - statistical analysis. -

    +

    By navigating to the sections in this topic, you can increase your knowledge in the areas + you've already looked at — installation, configuration, and N1QL — and + also learn about new and (in some cases) quite advanced topics; such as multi-node + clustering, failover, replication, and statistical analysis.

    - - Installation and Configuration - - -

    - Couchbase Server is supported on numerous popular operating systems. A complete list can be found - in the section Supported Platforms. - You may now wish to install Couchbase Server directly on one or more additional platforms: so, first, check - to ensure that the platforms are supported. -

    - -

    - Then, take a look at the information provided in - Installing On-Premises. This gives - you links to installation-procedures for each of the individual platforms. -

    - -

    - Couchbase Server can be run in a variety of configurations. You can perform configuration by means of the - Couchbase Web Console, via the Couchbase Command Line Interface (CLI), or with the Couchbase REST - API. See the section - Initialize the Cluster, for details. -

    - -

    - Periodically, you will also need to stop and start individual server-nodes. Information on doing this is - provided in the section - Couchbase Server Startup and Shutdown. -

    + Installation and Configuration + +

    Couchbase Server can be deployed and installed in several different ways, including on + traditional bare-metal servers, virtual machines, containers, and in the cloud. Take + a look at the information provided in to find the right + deployment for you.

    + +

    Couchbase Server can be configured in a variety of ways. You can perform the configuration + using the Couchbase Server Web Console, the Couchbase Command Line Interface (CLI), + or with the Couchbase REST API. See for details.

    + +

    Periodically, you will need to stop and start individual server nodes. Information on doing + this is provided in the section .

    - - Authorization - - -

    - Couchbase Server-resources are protected by mean of Role-Based - Access Control (RBAC). This means that different roles are assigned to - different users, each role being associated with a subset of - privileges on one or more resources. This makes it possible, for - example, for one user to be granted read-access on a particular - bucket, while another user is granted both - read and write access, on the same bucket. For - a detailed explanation, see - Authorization. -

    + Authorization + +

    Couchbase Server resources are protected by means of role-based access control (RBAC). + This means that different roles are assigned to different users, each + role being associated with a subset of privileges on one or more resources. + This makes it possible, for example, for one user to be granted read-access on a + particular bucket, while another user is granted both read and write-access on the + same bucket. For a detailed explanation, see .

    - - Querying with N1QL - - -

    - As you've already seen, you can query Couchbase Server manually, using the N1QL query language and the - Query Workbench provided - by the Couchbase Web Console. A full introduction to N1QL can be found in the section - Querying with N1QL: note that this - includes information on using cbq, which allows - N1QL queries to be entered at the command line. -

    - -

    - Additionally, N1QL queries can be specified by means of the Couchbase REST API. For information on doing - so, see the section - N1QL REST API -

    + Querying with N1QL + +

    As you've already seen, you can query Couchbase Server manually, using the N1QL query language + and the . A full + introduction to N1QL can be found in the section , including information on using + cbq, which allows N1QL queries to be entered at the command + line.

    + +

    Additionally, N1QL queries can be specified by means of the Couchbase REST API. For + information on doing so, see the section .

    - - Using the Couchbase SDK - - -

    - The Couchbase SDK exists in different versions: for node.js, Java, .NET, Go, - C, PHP, and Python. -

    - -

    - Under the DEVELOPERS tab in the vertical navigation-bar to the left, you'll find detailed information - on all aspects of the Couchbase SDK, and how to use it. First, take a look at the section - Start Using the SDK. Towards - the upper right of that page is a pull-down menu, which allows you to select a language. The page - for the language you select provides information on installing - supportive modules and libraries; as well as code-examples, to get you started with development. -

    + Using the Couchbase SDK + +

    The Couchbase SDK is available for several different programming languages. Take a look at the + section Start Using the SDK. You can select a language, and the page for that + language provides information on installing supportive modules and libraries, and + also includes code-examples to get you started with development.

    - - Concepts and Architecture - - -

    - Your hands-on progress with Couchbase Server will be greatly helped by a - good conceptual knowledge. - This is provided under the CONCEPTS & ARCHITECTURE tab, visible in the - vertical navigation-bar, to the left. Start by looking at the - Overview; then, go from there. - -

    + Concepts and Architecture + +

    Your hands-on progress with Couchbase Server will be greatly helped by a good + conceptual knowledge. Start by looking at the , and then go from there.

    - - Administration - - -

    - If you are an administrator, your priority will be to learn about system-setup and cluster-management. - So, access the documentation under the ADMINISTRATORS tab, visible in the navigation-bar; which - includes extensive information - on clusters, data-buckets, indexing, replication, security, and more. -

    + Administration + +

    If you are an administrator, your priority will be to learn about system setup and cluster + management. Start by reading an overview of administrative tasks in .

    - - Integration - - -

    - In some cases, you may wish to integrate Couchbase Server with another, different data-repository. For example, - you might wish to continue using your current Elasticsearch database for the performance of free text - searches; and extend these searches to Couchbase Server-data, so that Couchbase-documents can be retrieved. - For this to be possible, - data must be shared between the repositories, and your querying coordinated between them. -

    - -

    - Under the INTEGRATIONS - tab, you can find information about Elasticsearch and all other repositories with which you can integrate Couchbase Server, - thereby enriching your existing search-and-retrieval mechanisms. -

    + Integration + +

    In some cases, you may wish to integrate Couchbase Server with another, different + data-repository. For example, you might wish to continue using your current + Elasticsearch database for the performance of free text searches, and extend these + searches to Couchbase Server-data, so that Couchbase-documents can be retrieved. For + this to be possible, data must be shared between the repositories, and your querying + coordinated between them.

    + +

    See for information about + Elasticsearch and all other repositories with which you can integrate Couchbase + Server.

    - - Additional Information - - -

    - You can find extensive reference information on APIs, analytics, and tools under the REFERENCES tab, in the navigation-bar. - RELEASE NOTES can also be accessed there. -

    + Additional Reference Information + +

    When you start interacting with Couchbase in more advanced ways, make sure to explore some of + the reference documentation, such as and .

    diff --git a/content/getting-started/do-a-quick-install.dita b/content/getting-started/do-a-quick-install.dita index 73fbd842d9..e6ffdf31bb 100644 --- a/content/getting-started/do-a-quick-install.dita +++ b/content/getting-started/do-a-quick-install.dita @@ -5,217 +5,98 @@ - - Do a Quick Install - + Do a Quick Install - - First-time users can get Couchbase Server running simply and rapidly by using Docker. After it has been downloaded, - a single command is all you need; to obtain, install, configure, and run Couchbase Server. - + First-time users can get Couchbase Server running simply and rapidly by using Docker. + Once you install Docker, you can use a single command to download and install a + pre-configured version of Couchbase Server Enterprise Edition on your computer. - + + The Docker container image that is used in this topic is meant to set up a 'sandbox' + version of Couchbase Server. It's based on the Enterprise Edition and contains scripts + that automatically configure several Couchbase components during installation. Although + this particular image works great for a demo, it is not meant for production. For + information about using containers to run Couchbase Server in production, see .
    - - Download Docker - + Download and Install Docker -

    - Docker is a freely available application that lets you install a fully configured Couchbase Server - with a single command. - To download Docker, go to the - Docker - installation page. This displays an Install the Platform pane: -

    - -

    - -

    - -

    - Pick the version of Docker that is appropriate for your platform: Mac, Windows, or Linux. Then, left-click on the - corresponding Download or Install button: -

    - -
      - -
    • - On Mac, the Docker.dmg package is now downloaded. When the - download has completed, open the - package by double left-clicking on the icon. An interactive screen is then displayed: follow - the instruction it provides, and drag-and-drop Docker into your Applications folder. - -

      - -

      -
      • - -
    • - On Windows, when you left-click on the Download, - button, a .msi file is downloaded. When the download has - completed, open the file, and proceed as directed with installation. - -

      - -

      -
      • - -
    • - On Linux, a list of Docker-supported distributions is displayed. From this list, pick the - entry that corresponds to your current platform. You now see an instruction-page, provided by - Docker. Follow these instructions, to perform installation. - -

      - -

      -
      • -
    - -

    - Note: For installation, you may need supervisor privileges on your machine (obtained, for - example, by means of the sudo command, on MacOS and Linux). -

    +

    Docker is a freely available application that lets you install a fully-configured instance of + Couchbase Server with a single command. To set up Docker on your computer, go to the + Docker installation page and follow the instructions.

    - - Run Docker, to Install Couchbase Server - - + Use Docker to Install Couchbase Server +

    Once Docker is installed on your computer, you can go ahead and install Couchbase + Server.

    - First, bring up an interactive console on your screen. Then, against the console-prompt, enter the following command: + You may need administrative or root privileges on your computer to complete the + installation.

    - $ docker run -t --name db -p 8091-8094:8091-8094 -p 11210:11210 couchbase/sandbox:5.0.0-beta - -

    - Docker now obtains, downloads, installs, and configures Couchbase Server. During the download process, you see output - like the following, in your console: -

    +

    Open a console window on your computer and enter the following command:

    - $ docker run -t --name db -p 8091-8094:8091-8094 -p 11210:11210 couchbase/server:sandbox -Unable to find image 'couchbase/server:sandbox' locally -sandbox: Pulling from couchbase/server -862a3e9af0ae: Pull complete -6498e51874bf: Pull complete -159ebdd1959b: Pull complete -0fdbedd3771a: Pull complete -7a1f7116d1e3: Pull complete -b46618b72574: Pull complete -67ea61a8c297: Pull complete -969bb5073704: Pull complete -ef8eeaf0285d: Pull complete -1a797cd531fd: Pull complete -2667ee2d486a: Pull complete -d12025fc6fb6: Pull complete -fc670031c36d: Pull complete -da07033be0a7: Pull complete -Digest: sha256:02231569d6ac86c0f5a05bd53665e76472ccd515de460bb56f6e621c868b6305 -Status: Downloaded newer image for couchbase/sandbox:5.0.0-beta + docker run -t --name db -p 8091-8094:8091-8094 -p 11210:11210 couchbase/server-sandbox:5.5.0 -

    - Eventually, Couchbase Server is started, in a Docker virtual environment: -

    +

    When you run the command, Docker downloads, installs, and configures Couchbase Server. + Eventually, Couchbase Server is started in a Docker virtual environment:

    . . . -Starting Couchbase Server -- Web UI available at http://<ip>:8091 and logs available in /opt/couchbase/var/lib/couchbase/logs +Starting Couchbase Server -- Web UI available at http://<ip>:8091 +and logs available in /opt/couchbase/var/lib/couchbase/logs Configuring Couchbase Server. Please wait (~60 sec)... -Configuring index and memory quotas -Configuring Services -Creating default Administrator credentials -Enabling memory optimized indexes -Loading travel-sample bucket -Configuration completed - Couchbase Admin UI: http://localhost:8091 - - -

    - When start-up is thus concluded, Couchbase Server is running in a default configuration, and can be accessed. +Configuration completed! +Couchbase Admin UI: http://localhost:8091 +Login credentials: Administrator / password -

    - -

    - Note: From this point, while you are using Couchbase Server, don't close the console; since this - terminates both Docker and Couchbase Server. -

    +

    From this point onward, don't close the console while you're using Couchbase Server; this + terminates both Docker and Couchbase Server.

    - - Next - + Next -

    - You can now proceed directly to the next stage of the Getting Started sequence, and - Look at the Results - of your installation and configuration, using the Couchbase Web Console. -

    +

    You can now proceed to the next step: .

    - - Other Destinations - - -

    - At the end of the Getting Started sequence, you'll find a comprehensive list of links; which allow - you to investigate in greater detail all the subjects introduced on this page. However, - if you don't want to wait, you can learn more about Couchbase Server installation and - configuration right away, by visiting the following locations: -

    + Other Destinations
      -
    • - Installing On-Premises explains how - to install Couchbase Server directly onto your host, without the use of Docker or any other virtual - environment. This kind of direct install is the one you will certainly use for production-deployments, - and possibly for additional development and testing activities. - -

      - -

      -
      • +
    • : Explains how to install + Couchbase Server directly onto your host, without the use of Docker or any other + virtual environment. This kind of direct install is very common for + production-deployments, as well as development and testing activities.
      • -
    • - Initialize the Cluster provides a - detailed explanation of how to custom-configure Couchbase Server at start-up. This is the procedure you - will certainly use in production; as well as for testing different configurations. - The available options include use of the Couchbase Web Console UI, the Couchbase - REST API, and the Couchbase Command Line Interface. - -

      - -

      -
      • +
    • : Provides a detailed explanation of how + to custom-configure Couchbase Server at start-up. This is the procedure you will + certainly use in production; as well as for testing different configurations. + The available options include use of the Couchbase Server Web Console, the + Couchbase REST API, and the Couchbase Command Line Interface.
      • -
    • - Couchbase Server Startup - and Shutdown tells you how to start and stop the server, using commands specific to your underlying platform. -

      - -

      -
      • +
    • : Explains how to start and + stop the server using the commands that are specific to your underlying + platform.
    -

    - Optionally, when you have inspected these other destinations, - you can return to the current page, and - continue working through the Getting Started sequence. -

    -
    diff --git a/content/getting-started/images/ClusterOverview.png b/content/getting-started/images/ClusterOverview.png index f97bad71b5..d8a72df26e 100644 Binary files a/content/getting-started/images/ClusterOverview.png and b/content/getting-started/images/ClusterOverview.png differ diff --git a/content/getting-started/images/Octocat.png b/content/getting-started/images/Octocat.png new file mode 100644 index 0000000000..91057da496 Binary files /dev/null and b/content/getting-started/images/Octocat.png differ diff --git a/content/getting-started/images/airlin10document.png b/content/getting-started/images/airlin10document.png deleted file mode 100644 index 555d5feca2..0000000000 Binary files a/content/getting-started/images/airlin10document.png and /dev/null differ diff --git a/content/getting-started/images/applications-folder.png b/content/getting-started/images/applications-folder.png deleted file mode 100644 index e8077cde34..0000000000 Binary files a/content/getting-started/images/applications-folder.png and /dev/null differ diff --git a/content/getting-started/images/available-index-query.png b/content/getting-started/images/available-index-query.png deleted file mode 100644 index 3b28ac3ce8..0000000000 Binary files a/content/getting-started/images/available-index-query.png and /dev/null differ diff --git a/content/getting-started/images/brewery-query-1.png b/content/getting-started/images/brewery-query-1.png deleted file mode 100644 index b8e2116bb7..0000000000 Binary files a/content/getting-started/images/brewery-query-1.png and /dev/null differ diff --git a/content/getting-started/images/cbq-prompt.png b/content/getting-started/images/cbq-prompt.png deleted file mode 100644 index 4fc09357de..0000000000 Binary files a/content/getting-started/images/cbq-prompt.png and /dev/null differ diff --git a/content/getting-started/images/consoleLogin.png b/content/getting-started/images/consoleLogin.png index 815dc95e18..5d5fd24247 100644 Binary files a/content/getting-started/images/consoleLogin.png and b/content/getting-started/images/consoleLogin.png differ diff --git a/content/getting-started/images/couchbase-menu.png b/content/getting-started/images/couchbase-menu.png deleted file mode 100644 index ec684e1921..0000000000 Binary files a/content/getting-started/images/couchbase-menu.png and /dev/null differ diff --git a/content/getting-started/images/create-default-bucket-screen.png b/content/getting-started/images/create-default-bucket-screen.png deleted file mode 100644 index bfe3039b17..0000000000 Binary files a/content/getting-started/images/create-default-bucket-screen.png and /dev/null differ diff --git a/content/getting-started/images/dataBucketsTab.png b/content/getting-started/images/dataBucketsTab.png deleted file mode 100644 index 9709ac28bf..0000000000 Binary files a/content/getting-started/images/dataBucketsTab.png and /dev/null differ diff --git a/content/getting-started/images/docker-download-page.png b/content/getting-started/images/docker-download-page.png deleted file mode 100644 index d92517a724..0000000000 Binary files a/content/getting-started/images/docker-download-page.png and /dev/null differ diff --git a/content/getting-started/images/documentsButton.png b/content/getting-started/images/documentsButton.png deleted file mode 100644 index 57d809601b..0000000000 Binary files a/content/getting-started/images/documentsButton.png and /dev/null differ diff --git a/content/getting-started/images/editDocumentButton.png b/content/getting-started/images/editDocumentButton.png deleted file mode 100644 index f748d7f3e3..0000000000 Binary files a/content/getting-started/images/editDocumentButton.png and /dev/null differ diff --git a/content/getting-started/images/editDocumentDialog.png b/content/getting-started/images/editDocumentDialog.png new file mode 100644 index 0000000000..64a2d580d8 Binary files /dev/null and b/content/getting-started/images/editDocumentDialog.png differ diff --git a/content/getting-started/images/executeButton.png b/content/getting-started/images/executeButton.png deleted file mode 100644 index c3015fa9b4..0000000000 Binary files a/content/getting-started/images/executeButton.png and /dev/null differ diff --git a/content/getting-started/images/firstQuery.png b/content/getting-started/images/firstQuery.png index 90f28a8889..7c89a820e6 100644 Binary files a/content/getting-started/images/firstQuery.png and b/content/getting-started/images/firstQuery.png differ diff --git a/content/getting-started/images/keyspace-query.png b/content/getting-started/images/keyspace-query.png deleted file mode 100644 index 9a8f6c4899..0000000000 Binary files a/content/getting-started/images/keyspace-query.png and /dev/null differ diff --git a/content/getting-started/images/primary-index-query.png b/content/getting-started/images/primary-index-query.png deleted file mode 100644 index d688f14822..0000000000 Binary files a/content/getting-started/images/primary-index-query.png and /dev/null differ diff --git a/content/getting-started/images/queryButton.png b/content/getting-started/images/queryButton.png deleted file mode 100644 index 1a226736f2..0000000000 Binary files a/content/getting-started/images/queryButton.png and /dev/null differ diff --git a/content/getting-started/images/queryDisplayButtons.png b/content/getting-started/images/queryDisplayButtons.png deleted file mode 100644 index 0c680d8f28..0000000000 Binary files a/content/getting-started/images/queryDisplayButtons.png and /dev/null differ diff --git a/content/getting-started/images/queryNoWHERE.png b/content/getting-started/images/queryNoWHERE.png deleted file mode 100644 index 5fc8b24339..0000000000 Binary files a/content/getting-started/images/queryNoWHERE.png and /dev/null differ diff --git a/content/getting-started/images/queryResultsJSON.png b/content/getting-started/images/queryResultsJSON.png index e82fd6f42e..946c959953 100644 Binary files a/content/getting-started/images/queryResultsJSON.png and b/content/getting-started/images/queryResultsJSON.png differ diff --git a/content/getting-started/images/queryResultsJSONnoWHERE.png b/content/getting-started/images/queryResultsJSONnoWHERE.png deleted file mode 100644 index a838469fbf..0000000000 Binary files a/content/getting-started/images/queryResultsJSONnoWHERE.png and /dev/null differ diff --git a/content/getting-started/images/queryResultsTable.png b/content/getting-started/images/queryResultsTable.png deleted file mode 100644 index 08f80b4065..0000000000 Binary files a/content/getting-started/images/queryResultsTable.png and /dev/null differ diff --git a/content/getting-started/images/queryResultsTree.png b/content/getting-started/images/queryResultsTree.png deleted file mode 100644 index 8b0cb2c27e..0000000000 Binary files a/content/getting-started/images/queryResultsTree.png and /dev/null differ diff --git a/content/getting-started/images/queryWorkbench.png b/content/getting-started/images/queryWorkbench.png index 0e40272549..fa1d9dc7b7 100644 Binary files a/content/getting-started/images/queryWorkbench.png and b/content/getting-started/images/queryWorkbench.png differ diff --git a/content/getting-started/images/saveAsDialog.png b/content/getting-started/images/saveAsDialog.png new file mode 100644 index 0000000000..f898b98686 Binary files /dev/null and b/content/getting-started/images/saveAsDialog.png differ diff --git a/content/getting-started/images/secondary-index-query.png b/content/getting-started/images/secondary-index-query.png deleted file mode 100644 index c43b0c77b1..0000000000 Binary files a/content/getting-started/images/secondary-index-query.png and /dev/null differ diff --git a/content/getting-started/images/terminal-window.png b/content/getting-started/images/terminal-window.png deleted file mode 100644 index 675bdf4a08..0000000000 Binary files a/content/getting-started/images/terminal-window.png and /dev/null differ diff --git a/content/getting-started/images/travelSampleBucketScreen.png b/content/getting-started/images/travelSampleBucketScreen.png index 95d65dc93f..02432b8a44 100644 Binary files a/content/getting-started/images/travelSampleBucketScreen.png and b/content/getting-started/images/travelSampleBucketScreen.png differ diff --git a/content/getting-started/images/travelSampleBucketScreenInitial.png b/content/getting-started/images/travelSampleBucketScreenInitial.png index 4eafe89f89..d0d07fa74a 100644 Binary files a/content/getting-started/images/travelSampleBucketScreenInitial.png and b/content/getting-started/images/travelSampleBucketScreenInitial.png differ diff --git a/content/getting-started/installing-linux.dita b/content/getting-started/installing-linux.dita new file mode 100644 index 0000000000..0177a4e724 --- /dev/null +++ b/content/getting-started/installing-linux.dita @@ -0,0 +1,148 @@ + + + + Quick Installation and Setup + You can install Couchbase Server on Linux, Microsoft Windows, or OS X systems. These + streamlined instructions will help you get up and running right away in a nonproduction + environment so you can start learning how to use Couchbase Server. + + + + Use the quick installation and setup instructions on this page only to try + out Couchbase Server. If you are installing Couchbase Server for development, test, + or production purposes, set up Couchbase Server by following the detailed setup instructions + in the Installation Guide. + You can also read about sizing the Couchbase Server cluster in the blog How many nodes?. +
    + Download Couchbase Server +

    If you haven't already downloaded Couchbase Server, get it now from the Couchbase Server downloads page.

    +

    On Linux, you can download Couchbase Server directly using the wget command.

    +
    +
    + Install on Red Hat or CentOS Linux systems +

    To install on Red Hat Enterprise Linux (RHEL) or CentOS Linux:

    +
      +
    1. Log on to the server as root.
      2. +
    2. Enter the following command to install Couchbase Server, replacing + couchbase-server-version.rpm with the name of the file + you + downloaded:$ rpm --install couchbase-server-version.rpm

      Couchbase + Server starts automatically after the installation completes.

      3. +
    +
    + +
    + Install on Debian or Ubuntu Linux systems + +

    To install on Debian or Ubuntu Linux:

    +
      +
    1. Log on to the server as root.
      2. +
    2. Enter the following command to install OpenSSL, replacing version with a specific version + number of + OpenSSL:$ apt-get install libsslversion
      3. +
    3. Enter the following command to install Couchbase Server, replacing + couchbase-server-version.deb with the name of the file + you + downloaded:$ dpkg -i couchbase-server-version.deb

      Couchbase + Server starts automatically after the installation completes.

      4. + +
    + +
    + +
    + Install on OS X systems +

    If you have an existing installation:

    +
      +
    1. Shut down any running instances of Couchbase Server.
      2. +
    2. Delete any previous installations either by using the command + line or by dragging the Couchbase Server application to the Trash.
      3. +
    3. Open a Terminal window and delete any application support files from previous + installations: +$ rm -rf ~/Library/Application\Support/Couchbase +$ rm -rf ~/Library/Application\Support/Membase
      4. +
    +

    To install Couchbase Server:

    +
      +
    1. Download the zip file with the Couchbase Server package.
      2. +
    2. Expand the downloaded zip file (if it didn't expand automatically) and open the folder. +

      The folder contains the Couchbase Server application and + README.txt file.

      3. +
    3. Move the Couchbase Server file to the Applications + folder.
      4. +
    4. Start Couchbase Server by double-clicking the application icon.

      If you get a message that + says you can't run Couchbase Server due to security restrictions, + right-click the icon for Couchbase Server and choose + Open from the pop-up menu.

      5. +
    + +
    +
    + Install on 64-bit Microsoft Windows systems +

    To install on Microsoft Windows:

    +
      +
    1. Double-click the downloaded executable file to start the installer.
      2. +
    2. Follow the installer prompts. +
        +
      • If desired, you can change the installation location.
        • +
      • If the default port is unavailable, the installer prompts you for a different + port to use for server administration.
        • +
      • When the installer asks if you want to increase the number of ephemeral ports, + click Yes.
        • +
      3. +
    3. After the installation is complete, restart Couchbase Server to apply the port changes.
      4. +
    +
    + + +
    + Set up Couchbase Server + +

    Set up Couchbase Server when you start the Couchbase Web Console for the first time.

    + Use the quick installation and setup instructions on this page only to try + out Couchbase Server. If you are installing Couchbase Server for development, test, + or production purposes, set up Couchbase Server by following the detailed setup instructions + in the Installation Guide. + +

    To set up Couchbase Server in a non production environment, you can accept the default values + provided on most of the set-up screens.

    +
      +
    1. Open a browser and navigate to http://hostname:8091/.
      2. +
    2. In the URL, + hostname represents the name or IP address of the + computer that hosts Couchbase Server. If Couchbase Server is running + locally, enter localhost for the host name.
      3. +
    3. Click Setup.
      4. +
    4. On the Configure Server screen (Step 1 of 5), click + Next to accept the default values for a new + cluster.
      5. +
    5. On the Sample Buckets screen (Step 2 of 5), under Available + Samples select the two samples we will use later in this + tutorial: beer-sample and + travel-sample and click + Next.
      6. +
    6. On the Create Default Bucket screen (Step 3 of 5), under Memory Size + set the Per Node RAM Quota to 100 MB and click + Next.
      7. +
    7. On the Notifications screen (Step 4 of 5), enter your registration + information, agree to the terms and conditions, and click + Next.
      8. +
    8. On the Configure Server screen (Step 5 of 5), enter and verify a + password for the administrator account, and click + Next.

      The Couchbase Web Console opens and displays + the Cluster Overview.

      9. +
    +

    While the Couchbase Web Console is open, click on the tabs in the top menu and look at the + screens to familiarize yourself with the information and available options.

    +
    + +     +     diff --git a/content/getting-started/installing-osx.dita b/content/getting-started/installing-osx.dita new file mode 100644 index 0000000000..0177a4e724 --- /dev/null +++ b/content/getting-started/installing-osx.dita @@ -0,0 +1,148 @@ + + + + Quick Installation and Setup + You can install Couchbase Server on Linux, Microsoft Windows, or OS X systems. These + streamlined instructions will help you get up and running right away in a nonproduction + environment so you can start learning how to use Couchbase Server. + + + + Use the quick installation and setup instructions on this page only to try + out Couchbase Server. If you are installing Couchbase Server for development, test, + or production purposes, set up Couchbase Server by following the detailed setup instructions + in the Installation Guide. + You can also read about sizing the Couchbase Server cluster in the blog How many nodes?. +
    + Download Couchbase Server +

    If you haven't already downloaded Couchbase Server, get it now from the Couchbase Server downloads page.

    +

    On Linux, you can download Couchbase Server directly using the wget command.

    +
    +
    + Install on Red Hat or CentOS Linux systems +

    To install on Red Hat Enterprise Linux (RHEL) or CentOS Linux:

    +
      +
    1. Log on to the server as root.
      2. +
    2. Enter the following command to install Couchbase Server, replacing + couchbase-server-version.rpm with the name of the file + you + downloaded:$ rpm --install couchbase-server-version.rpm

      Couchbase + Server starts automatically after the installation completes.

      3. +
    +
    + +
    + Install on Debian or Ubuntu Linux systems + +

    To install on Debian or Ubuntu Linux:

    +
      +
    1. Log on to the server as root.
      2. +
    2. Enter the following command to install OpenSSL, replacing version with a specific version + number of + OpenSSL:$ apt-get install libsslversion
      3. +
    3. Enter the following command to install Couchbase Server, replacing + couchbase-server-version.deb with the name of the file + you + downloaded:$ dpkg -i couchbase-server-version.deb

      Couchbase + Server starts automatically after the installation completes.

      4. + +
    + +
    + +
    + Install on OS X systems +

    If you have an existing installation:

    +
      +
    1. Shut down any running instances of Couchbase Server.
      2. +
    2. Delete any previous installations either by using the command + line or by dragging the Couchbase Server application to the Trash.
      3. +
    3. Open a Terminal window and delete any application support files from previous + installations: +$ rm -rf ~/Library/Application\Support/Couchbase +$ rm -rf ~/Library/Application\Support/Membase
      4. +
    +

    To install Couchbase Server:

    +
      +
    1. Download the zip file with the Couchbase Server package.
      2. +
    2. Expand the downloaded zip file (if it didn't expand automatically) and open the folder. +

      The folder contains the Couchbase Server application and + README.txt file.

      3. +
    3. Move the Couchbase Server file to the Applications + folder.
      4. +
    4. Start Couchbase Server by double-clicking the application icon.

      If you get a message that + says you can't run Couchbase Server due to security restrictions, + right-click the icon for Couchbase Server and choose + Open from the pop-up menu.

      5. +
    + +
    +
    + Install on 64-bit Microsoft Windows systems +

    To install on Microsoft Windows:

    +
      +
    1. Double-click the downloaded executable file to start the installer.
      2. +
    2. Follow the installer prompts. +
        +
      • If desired, you can change the installation location.
        • +
      • If the default port is unavailable, the installer prompts you for a different + port to use for server administration.
        • +
      • When the installer asks if you want to increase the number of ephemeral ports, + click Yes.
        • +
      3. +
    3. After the installation is complete, restart Couchbase Server to apply the port changes.
      4. +
    +
    + + +
    + Set up Couchbase Server + +

    Set up Couchbase Server when you start the Couchbase Web Console for the first time.

    + Use the quick installation and setup instructions on this page only to try + out Couchbase Server. If you are installing Couchbase Server for development, test, + or production purposes, set up Couchbase Server by following the detailed setup instructions + in the Installation Guide. + +

    To set up Couchbase Server in a non production environment, you can accept the default values + provided on most of the set-up screens.

    +
      +
    1. Open a browser and navigate to http://hostname:8091/.
      2. +
    2. In the URL, + hostname represents the name or IP address of the + computer that hosts Couchbase Server. If Couchbase Server is running + locally, enter localhost for the host name.
      3. +
    3. Click Setup.
      4. +
    4. On the Configure Server screen (Step 1 of 5), click + Next to accept the default values for a new + cluster.
      5. +
    5. On the Sample Buckets screen (Step 2 of 5), under Available + Samples select the two samples we will use later in this + tutorial: beer-sample and + travel-sample and click + Next.
      6. +
    6. On the Create Default Bucket screen (Step 3 of 5), under Memory Size + set the Per Node RAM Quota to 100 MB and click + Next.
      7. +
    7. On the Notifications screen (Step 4 of 5), enter your registration + information, agree to the terms and conditions, and click + Next.
      8. +
    8. On the Configure Server screen (Step 5 of 5), enter and verify a + password for the administrator account, and click + Next.

      The Couchbase Web Console opens and displays + the Cluster Overview.

      9. +
    +

    While the Couchbase Web Console is open, click on the tabs in the top menu and look at the + screens to familiarize yourself with the information and available options.

    +
    + +     +     diff --git a/content/getting-started/installing-windows.dita b/content/getting-started/installing-windows.dita new file mode 100644 index 0000000000..0177a4e724 --- /dev/null +++ b/content/getting-started/installing-windows.dita @@ -0,0 +1,148 @@ + + + + Quick Installation and Setup + You can install Couchbase Server on Linux, Microsoft Windows, or OS X systems. These + streamlined instructions will help you get up and running right away in a nonproduction + environment so you can start learning how to use Couchbase Server. + + + + Use the quick installation and setup instructions on this page only to try + out Couchbase Server. If you are installing Couchbase Server for development, test, + or production purposes, set up Couchbase Server by following the detailed setup instructions + in the Installation Guide. + You can also read about sizing the Couchbase Server cluster in the blog How many nodes?. +
    + Download Couchbase Server +

    If you haven't already downloaded Couchbase Server, get it now from the Couchbase Server downloads page.

    +

    On Linux, you can download Couchbase Server directly using the wget command.

    +
    +
    + Install on Red Hat or CentOS Linux systems +

    To install on Red Hat Enterprise Linux (RHEL) or CentOS Linux:

    +
      +
    1. Log on to the server as root.
      2. +
    2. Enter the following command to install Couchbase Server, replacing + couchbase-server-version.rpm with the name of the file + you + downloaded:$ rpm --install couchbase-server-version.rpm

      Couchbase + Server starts automatically after the installation completes.

      3. +
    +
    + +
    + Install on Debian or Ubuntu Linux systems + +

    To install on Debian or Ubuntu Linux:

    +
      +
    1. Log on to the server as root.
      2. +
    2. Enter the following command to install OpenSSL, replacing version with a specific version + number of + OpenSSL:$ apt-get install libsslversion
      3. +
    3. Enter the following command to install Couchbase Server, replacing + couchbase-server-version.deb with the name of the file + you + downloaded:$ dpkg -i couchbase-server-version.deb

      Couchbase + Server starts automatically after the installation completes.

      4. + +
    + +
    + +
    + Install on OS X systems +

    If you have an existing installation:

    +
      +
    1. Shut down any running instances of Couchbase Server.
      2. +
    2. Delete any previous installations either by using the command + line or by dragging the Couchbase Server application to the Trash.
      3. +
    3. Open a Terminal window and delete any application support files from previous + installations: +$ rm -rf ~/Library/Application\Support/Couchbase +$ rm -rf ~/Library/Application\Support/Membase
      4. +
    +

    To install Couchbase Server:

    +
      +
    1. Download the zip file with the Couchbase Server package.
      2. +
    2. Expand the downloaded zip file (if it didn't expand automatically) and open the folder. +

      The folder contains the Couchbase Server application and + README.txt file.

      3. +
    3. Move the Couchbase Server file to the Applications + folder.
      4. +
    4. Start Couchbase Server by double-clicking the application icon.

      If you get a message that + says you can't run Couchbase Server due to security restrictions, + right-click the icon for Couchbase Server and choose + Open from the pop-up menu.

      5. +
    + +
    +
    + Install on 64-bit Microsoft Windows systems +

    To install on Microsoft Windows:

    +
      +
    1. Double-click the downloaded executable file to start the installer.
      2. +
    2. Follow the installer prompts. +
        +
      • If desired, you can change the installation location.
        • +
      • If the default port is unavailable, the installer prompts you for a different + port to use for server administration.
        • +
      • When the installer asks if you want to increase the number of ephemeral ports, + click Yes.
        • +
      3. +
    3. After the installation is complete, restart Couchbase Server to apply the port changes.
      4. +
    +
    + + +
    + Set up Couchbase Server + +

    Set up Couchbase Server when you start the Couchbase Web Console for the first time.

    + Use the quick installation and setup instructions on this page only to try + out Couchbase Server. If you are installing Couchbase Server for development, test, + or production purposes, set up Couchbase Server by following the detailed setup instructions + in the Installation Guide. + +

    To set up Couchbase Server in a non production environment, you can accept the default values + provided on most of the set-up screens.

    +
      +
    1. Open a browser and navigate to http://hostname:8091/.
      2. +
    2. In the URL, + hostname represents the name or IP address of the + computer that hosts Couchbase Server. If Couchbase Server is running + locally, enter localhost for the host name.
      3. +
    3. Click Setup.
      4. +
    4. On the Configure Server screen (Step 1 of 5), click + Next to accept the default values for a new + cluster.
      5. +
    5. On the Sample Buckets screen (Step 2 of 5), under Available + Samples select the two samples we will use later in this + tutorial: beer-sample and + travel-sample and click + Next.
      6. +
    6. On the Create Default Bucket screen (Step 3 of 5), under Memory Size + set the Per Node RAM Quota to 100 MB and click + Next.
      7. +
    7. On the Notifications screen (Step 4 of 5), enter your registration + information, agree to the terms and conditions, and click + Next.
      8. +
    8. On the Configure Server screen (Step 5 of 5), enter and verify a + password for the administrator account, and click + Next.

      The Couchbase Web Console opens and displays + the Cluster Overview.

      9. +
    +

    While the Couchbase Web Console is open, click on the tabs in the top menu and look at the + screens to familiarize yourself with the information and available options.

    +
    + +     +     diff --git a/content/getting-started/look-at-the-results.dita b/content/getting-started/look-at-the-results.dita index b6f98a5584..18f1cb619e 100644 --- a/content/getting-started/look-at-the-results.dita +++ b/content/getting-started/look-at-the-results.dita @@ -5,184 +5,166 @@ - - Look at the Results - + Explore the Server Configuration - - Once you have Couchbase Server running, you can log into the Couchbase Web Console, and start - to examine the different features that it provides. These include a facility for inspecting Couchbase - documents, organized within buckets. - + Once you have Couchbase Server running, you can log into the Couchbase Server Web + Console, and start to examine the different features that it provides. These + features include an interface for inspecting Couchbase documents, organized within + buckets.
    - - Access the Console and Log in - + Access the Console and Log In -

    - The default location for the Couchbase Web Console (whether it is running, as here, in - a Docker virtual environment, or directly on your platform) is - localhost:8091. So, type this into the address-bar of your browser, and - hit return. The following log-in screen is displayed: -

    +

    The default address for the Couchbase Server Web Console (whether it is running in a Docker + virtual environment, or directly on your platform) is + localhost:8091. Enter the address into the address bar of a + supported Web + browser, and hit return. The Couchbase Server log-in screen is displayed:

    -

    - The credentials you now enter are ones that have been established by default, by Docker. - Your username is Administrator, - and your password is password. Type these into the appropriate fields, and again - hit return. -

    +

    The Docker image that you installed comes pre-configured with a default username and + password:

      +
    • Username: Administrator
      • +
    • Password: password
      • +
    Type these credentials into the appropriate fields and click Sign + In.

    -
    - - - Examine the Cluster Dashboard - - -

    - You now see the following screen: -

    - -

    - -

    - -

    - This is the Cluster Dashboard screen, which provides a graphical summary - of the current state of your Couchbase cluster. The term cluster might seem unexpected at this - point, since you are only running a single instance of Couchbase Server: but that nevertheless does count as a - cluster of one. -

    - -

    - All the values displayed on this screen were configured by default, when you installed Couchbase Server - by means of Docker. In production, you will specify these values individually, according to - your needs. -

    - -

    - Notice the Buckets panel, towards the bottom. This shows that you have a single active - bucket on the system — bucket meaning a logical group of data-items. Taking a closer look - at this bucket will give you some idea of how Couchbase stores data, and so prepare your way for - making data-queries. So, left-click on the Buckets tab, - on the vertical control-bar, at the left-hand side of the console: -

    - -

    - -

    - -

    - This now brings up the Buckets screen. -

    - -
    +
    Examine the Cluster Dashboard

    Every time you log into the Web Console, + you are brought to the Cluster Dashboard:

    + +

    The Cluster Dashboard provides a graphical summary of the + current state of your Couchbase cluster. The term cluster might seem unexpected + at this point, since you are only running a single instance of Couchbase Server; but + nevertheless, it counts as a cluster of one.

    All of the values that are displayed on + this screen were automatically configured by the sandbox container image during + installation. In production, you will specify these values individually according to + your needs.

    Notice the Buckets panel, towards the + bottom. This shows that you have a single active bucket on the system — + bucket meaning a logical group of data-items. Taking a closer look at + this bucket will give you some idea of how Couchbase stores data, and prepare you to + make your first data-queries.

    - - - Examine your Bucket and its Documents - - -

    - The Buckets screen provides information on the single bucket you currently - have on the system: -

    - -

    - -

    - -

    - The name of the single bucket, travel-sample appears towards the left of the single row. Additional - information is provided in columns, across the row. -

    - -

    - To inspect the individual documents contained within this bucket, left-click on the Documents - button, which is towards the right of the travel-sample - row. -

    - -

    - -

    - -

    - The display now changes to the following: -

    - -

    - -

    - -

    - This shows, in a succession of page-views, the Couchbase documents that are contained within the bucket. Since, in the - next stages of the - Getting Started sequence, - you'll perform queries, to pull out specific items and elements, it's worthwhile now to take a look at the structure of - a document. So, left-click on the Edit Document button, towards the right of the first row: -

    - -

    - -

    - -

    - The screen now changes, to show the individual document for the row on which you clicked: -

    - -

    - -

    - -

    - Look at the structure of the JSON document now displayed: it consists of a series of name-value - (or, as they are sometimes expressed, key-value) pairs. By referencing a name, you can find - its value in one or more documents, by means of a query. For example, here, if you searched on - the name country, you would return the value United States; if on - the name icao, the value MLA. -

    - + Examine Your Bucket and Its Documents +

    Click Buckets in the left-hand navigation bar to bring up the + Buckets screen.

    +

    The Buckets screen shows that you have a single active bucket + on the system (bucket meaning a logical group of data-items). Taking a closer + look at this bucket will give you some idea of how Couchbase stores data, and + prepare you to make your first data-queries:

    +

    + +

    +

    The name of the single bucket (travel-sample) appears towards the + left of the single row. Additional information is provided in columns, across the + row.

    +

    Click Documents, which is located towards the right of the + travel-sample row, to go to the + Documents screen.

    +

    The Documents screen lets you inspect the individual documents that are + contained within the bucket:

    +

    + +

    +

    The Documents screen shows, in a succession of page-views, the + documents that are contained within the bucket. The following document retrieval + controls are provided:

    +
      +
    • Bucket: A drop-down menu that displays the name of the bucket whose documents are + currently being viewed. You can use the drop-down menu to select other available + buckets.
      • +
    • Limit: The maximum number of rows (documents) to retrieve and display at + once.
      • +
    • Offset: The number of documents in the entire set of the current bucket that should + be skipped, before display begins. Notice that when you click Next + Batch >, the Offset increases by the same + value that is specified in Limit.
      • +
    • Document ID: Accepts the ID of a specific document. Leave this field blank to + retrieve documents based on Limit and Offset.
      • +
    • Where: Accepts a N1QL query — specifically a WHERE clause — which + determines the subset of documents to be displayed. (You will learn more about + N1QL in a later step of this Getting Started sequence.)
      • +
    +

    In the results, each document is represented by an individual row that contains its ID and a + summary of its contents. You can switch between two views: + simple and spreadsheet. In the + spreadsheet view, you can edit the document fields + directly, since each key has its own column in which the corresponding value for + each document is provided, row by row.

    +

    The following buttons appear on the left side of each row:

    +
      +
    • Edit document as JSON: Click this button to bring up the Edit + Document dialog, which allows you to make direct edits to the document:

      + +

      The document consists of a series of key-value pairs (or, as they + are sometimes expressed, name-value pairs). You can make + modifications to key-values directly in this editor. As will be demonstrated + later, Couchbase Server allows you to search for keys, and return the + corresponding values, by means of a query. For example, here, if you + searched on the name country, you would return the value + United States; if on the name icao, + the value MLA.

      If you make changes in the + Edit Document dialog, click + Save to save your changes. If you want to create + a new document based on an existing document, you can click the Make a + copy of this document button (described next). If you want to create + an entirely new document, you can click the ADD + DOCUMENT button in the upper-right.

      • +
    • Make a copy of this document: Click this button to bring up the Save + As dialog, which allows you to create a new document based on + the existing one:

      + +

      Either click the Save button to save the copy under + the placeholder name that is provided, or edit the placeholder-name before + saving.

      Note that if you are using the spreadsheet + view, any unsaved changes that you have made to document key-values will be + saved in the copied document and not the original, much like the behavior of + traditional text editing software.

      • +
    • Delete this document: Click this button to delete the document.
      • +
    • Save changes to document: If you make changes in the + spreadsheet view, this button becomes active for the + current row. Click it to save your changes to the document.
      • +
    +

    To view successive sets of documents, use the Next Batch > and + < Prev Batch buttons.

    - - Next - + Next -

    - Now that you have a basic familiarity with the way in which Couchbase Server organizes data, you can - start to define and execute queries, in order to return specific data-subsets. You'll experiment - with this in the next section, - Run Your First N1QL Query. -

    +

    Now that you have a basic familiarity with the way in which Couchbase Server organizes data, + you can start to define and execute queries to return specific data subsets. You'll + experiment with this in the next section: Run Your First N1QL Query.

    - - Other Destinations - + Other Destinations

    - To learn more about the Couchbase data-model, take a look at the introduction to - Data Modeling Basics. - This provides an extensive conceptual account of Couchbase buckets. Additional, practical information - can be found in the section - Data Bucket Setup. +

      +
    • : Contains + basic information about buckets.
      • +
    • : + Provides more information about the Couchbase data-model and an extensive + conceptual account of Couchbase buckets.
      • +

    diff --git a/content/getting-started/quick-install.dita b/content/getting-started/quick-install.dita new file mode 100644 index 0000000000..7024a3ac7d --- /dev/null +++ b/content/getting-started/quick-install.dita @@ -0,0 +1,148 @@ + + + + Quick Installation + Want to take a quick tour of the product? You're at the right place. You can choose to + launch a pre-configured Couchbase Docker container or perform a quick installation on the OS + of your choice. This topic helps you get up and running quickly in a non-production + environment so you can spend time exploring the various features that Couchbase offers. + + + + +
    + Download Couchbase Server +

    If you haven't already downloaded Couchbase Server, get it now from the Couchbase Server downloads page.

    +

    On Linux, you can download Couchbase Server directly using the wget command.

    +
    +
    + Install on Red Hat or CentOS Linux systems +

    To install on Red Hat Enterprise Linux (RHEL) or CentOS Linux:

    +
      +
    1. Log on to the server as root.
      2. +
    2. Enter the following command to install Couchbase Server, replacing + couchbase-server-version.rpm with the name of the file + you + downloaded:$ rpm --install couchbase-server-version.rpm

      Couchbase + Server starts automatically after the installation completes.

      3. +
    +
    + +
    + Install on Debian or Ubuntu Linux systems + +

    To install on Debian or Ubuntu Linux:

    +
      +
    1. Log on to the server as root.
      2. +
    2. Enter the following command to install OpenSSL, replacing version with a specific version + number of + OpenSSL:$ apt-get install libsslversion
      3. +
    3. Enter the following command to install Couchbase Server, replacing + couchbase-server-version.deb with the name of the file + you + downloaded:$ dpkg -i couchbase-server-version.deb

      Couchbase + Server starts automatically after the installation completes.

      4. + +
    + +
    + +
    + Install on OS X systems +

    If you have an existing installation:

    +
      +
    1. Shut down any running instances of Couchbase Server.
      2. +
    2. Delete any previous installations either by using the command + line or by dragging the Couchbase Server application to the Trash.
      3. +
    3. Open a Terminal window and delete any application support files from previous installations: + $ rm -rf ~/Library/Application\ Support/Couchbase +$ rm -rf ~/Library/Application\ Support/Membase
      4. +
    +

    To install Couchbase Server:

    +
      +
    1. Download the zip file with the Couchbase Server package.
      2. +
    2. Expand the downloaded zip file (if it didn't expand automatically) and open the folder. +

      The folder contains the Couchbase Server application and + README.txt file.

      3. +
    3. Move the Couchbase Server file to the Applications + folder.
      4. +
    4. Start Couchbase Server by double-clicking the application icon.

      If you get a message that + says you can't run Couchbase Server due to security restrictions, + right-click the icon for Couchbase Server and choose + Open from the pop-up menu.

      5. +
    + +
    +
    + Install on 64-bit Microsoft Windows systems +

    To install on Microsoft Windows:

    +
      +
    1. Double-click the downloaded executable file to start the installer.
      2. +
    2. Follow the installer prompts. +
        +
      • If desired, you can change the installation location.
        • +
      • If the default port is unavailable, the installer prompts you for a different + port to use for server administration.
        • +
      • When the installer asks if you want to increase the number of ephemeral ports, + click Yes.
        • +
      3. +
    3. After the installation is complete, restart Couchbase Server to apply the port changes.
      4. +
    +
    + + +
    + Set up Couchbase Server + +

    Set up Couchbase Server when you start the Couchbase Web Console for the first time.

    + Use the quick installation and setup instructions on this page only to try + out Couchbase Server. If you are installing Couchbase Server for development, test, + or production purposes, set up Couchbase Server by following the detailed setup instructions + in the Installation Guide. + +

    To set up Couchbase Server in a non production environment, you can accept the default values + provided on most of the set-up screens.

    +
      +
    1. Open a browser and navigate to http://hostname:8091/.
      2. +
    2. In the URL, + hostname represents the name or IP address of the + computer that hosts Couchbase Server. If Couchbase Server is running + locally, enter localhost for the host name.
      3. +
    3. Click Setup.
      4. +
    4. On the Configure Server screen (Step 1 of 5), click + Next to accept the default values for a new + cluster.
      5. +
    5. On the Sample Buckets screen (Step 2 of 5), under Available + Samples select the two samples we will use later in this + tutorial: beer-sample and + travel-sample and click + Next.
      6. +
    6. On the Create Default Bucket screen (Step 3 of 5), under Memory Size + set the Per Node RAM Quota to 100 MB and click + Next.
      7. +
    7. On the Notifications screen (Step 4 of 5), enter your registration + information, agree to the terms and conditions, and click + Next.
      8. +
    8. On the Configure Server screen (Step 5 of 5), enter and verify a + password for the administrator account, and click + Next.

      The Couchbase Web Console opens and displays + the Cluster Overview.

      9. +
    +

    While the Couchbase Web Console is open, click on the tabs in the top menu and look at the + screens to familiarize yourself with the information and available options.

    +
    + +     +     diff --git a/content/getting-started/start-here.dita b/content/getting-started/start-here.dita index f8d70b2c45..ae7d37a95d 100644 --- a/content/getting-started/start-here.dita +++ b/content/getting-started/start-here.dita @@ -1,106 +1,61 @@ - - - - - Start Here! - - - - If you are a first-time user, you can get a great, quick introduction to using Couchbase Server, just by reading - this section. In a few brief steps, you obtain direct experience of installing, running, and performing - interactive queries on the server. - - - - + Start Here! + If you are a first-time user, you can get a quick introduction to using Couchbase + Server just by reading this section. In a few brief steps, you'll get direct experience with + installing, running, and performing interactive queries on the server. +
    - - - Steps for First-Time Users - - + Steps for First-Time Users +

    The following steps will guide you through some of the key aspects of Couchbase Server. At the + conclusion of the Getting Started sequence, you'll have reached an excellent + starting point for further, more detail-driven activities.

    - You'll now be guided through some of the key aspects of Couchbase Server. At the conclusion of the sequence, - you'll have attained an excellent starting-point for further, more detail-driven activities. The - steps are as follows: + + Getting Started Sequence for First-Time Users + + + + + + Step + Topic + + + + + Step 1 + + + + Step 2 + + + + Step 3 + + + + Step 4 + + + + +

    - -
      - -
    1. - Do a Quick Install - -

      - -

      -
      2. - -
    2. - Look at the Results - -

      - -

      -
      3. - -
    3. - Run Your First N1QL Query - -

      - -

      -
      4. - - - -
    4. - Choose Your Next Steps - -

      - -

      -
      5. -
    - -

    - If, at any stage, you decide that you wish to - depart from the sequence, and make an immediate deep-dive into a specific topic, you can do so: by means - of links to Other Destinations. -

    - -

    - Alternatively, if you do elect to - work continuously through the whole sequence (as indeed we suggest), you ultimately arrive at the - Choose Your Next Steps - section; which provides a comprehensive list of options for gaining further - knowledge of Couchbase server. -

    - -

    - Now proceed to - Do a Quick Install. -

    - +

    If at any step you decide that you wish to depart from the sequence and make an + immediate deep-dive into a specific subject, each page has a section of Other + Destinations that contain links and more information.

    +

    Alternatively, if you elect to work continuously through the whole sequence (as indeed we + suggest), you ultimately arrive at the topic, which provides a + comprehensive list of options for gaining further knowledge about Couchbase + Server.

    - -     - - - - diff --git a/content/getting-started/starter-kits-latest-Rob.html b/content/getting-started/starter-kits-latest-Rob.html new file mode 100644 index 0000000000..6aff237f0b --- /dev/null +++ b/content/getting-started/starter-kits-latest-Rob.html @@ -0,0 +1,131 @@ + + + + + + +Starter Kits for Couchbase Developers + + + + + + + + + +
    + + + + + +
    +
    +

    Couchbase with Node.js using Express Framework

    +
    +
    +

    Create a "hello world" Node.js web application that uses Couchbase as the backend database.

    +

    Get the kit on Github

    +
    + +
    +
    +

    Couchbase with ASP.NET MVC

    +
    +
    +

    Create a "hello world" ASP.NET MVC web application that uses Couchbase as the backend database.

    +

    Get the kit on Github

    +
    + +
    +
    +

    Couchbase with Python SDK

    +
    +
    +

    Create a simple Python application to store and retrieve a JSON document in Couchbase.

    +

    Get the kit on Github

    +
    + +
    +
    +

    Couchbase Lite (CBL) on Android

    +
    +
    +

    Build an Android application consisting of a single activity that performs CRUD operations and uses Couchbase Lite to store documents locally on the device.

    +

    Get the kit on Github

    +
    + +
    +
    +

    Write Your Own Starter Kit

    +
    +
    +

    Use these guidelines to build your own guide using ASCIIdoc and push it to the couchbase-guides Github repository.

    +

    Get Kit on Github

    +
    + +
    + + + + + diff --git a/content/getting-started/starter-kits.html b/content/getting-started/starter-kits.html index f6a11529f6..bfded61af0 100644 --- a/content/getting-started/starter-kits.html +++ b/content/getting-started/starter-kits.html @@ -69,7 +69,7 @@

    Starter Kits for Couchbase Developers

    } } - +

    Couchbase Server in Docker

    @@ -90,12 +90,21 @@

    Couchbase Server on Amazon EC2

    -

    Couchbase Server with ASP.NET MVC

    +

    Couchbase Server with ASP.NET Core

    -

    Create a "hello world" ASP.NET MVC web application that uses Couchbase Server as the backend database.

    -

    Learn more

    +

    Create a "hello world" ASP.NET Core web application that uses Couchbase Server as the backend database.

    +

    Learn more

    + +
    +
    +

    Couchbase Server with ASP.NET WebAPI

    +
    +
    +

    Create a "hello world" ASP.NET WebAPI application that uses Couchbase Server as the backend database.

    +

    Learn more

    +
    @@ -105,7 +114,7 @@

    Couchbase with Java SDK

    Create a Java application that uses Couchbase Server as the backend to store a JSON document.

    Learn more

    - +

    Couchbase with Node.js using Express Framework

    @@ -158,7 +167,7 @@

    Couchbase Restore

    Learn more

    - +

    Write Your Own Starter Kit

    diff --git a/content/getting-started/try-a-query.dita b/content/getting-started/try-a-query.dita index ac7b27560f..9eb03782ba 100644 --- a/content/getting-started/try-a-query.dita +++ b/content/getting-started/try-a-query.dita @@ -5,54 +5,38 @@ - - Run Your First N1QL Query - + Run Your First N1QL Query - - N1QL (pronounced "nickel") is the Couchbase Server query language. - + Now that you have a basic understanding of buckets and documents, you can try querying + them using N1QL (pronounced "nickel"), the Couchbase Server query language.
    - - About N1QL - + About N1QL -

    - N1QL embraces the JSON document model and uses SQL-like syntax. In N1QL, you operate on JSON - documents, and the result of your operation is another JSON document. N1QL queries can - be run from the command line, using the cbq - tool; or by means of the Query Workbench, provided by the Couchbase Web Console. -

    +

    N1QL embraces the JSON document model and uses SQL-like syntax. In N1QL, you operate on JSON + documents, and the result of your operation is another JSON document. You can run + N1QL queries from the command line, using the cbq tool; or from the + Query Workbench in the Couchbase Server Web Console.

    -

    - A basic N1QL query has the following parts: -

    +

    A basic N1QL query has the following parts:

      -
    • - SELECT — The fields of each document to return.
      • -
    • - FROM — The data bucket in which to look.
      • -
    • - WHERE — The conditions that the document must satisfy.
      • +
    • SELECT — The fields of each document to return.
      • +
    • FROM — The data bucket in which to look.
      • +
    • WHERE — The conditions that the document must satisfy.
    -

    - Here's an example of a basic N1QL query and the JSON document it returns. - The query asks for the country associated with the airline - Excel Airways — note that for all identifiers (bucket names) that contain a hyphen character, you - need to enclose the name with backtick (`) characters: -

    +

    Here's an example of a basic N1QL query and the JSON document it returns. The query asks for + the country that is associated with the airline Excel Airways:

    SELECT country FROM `travel-sample` WHERE name = "Excel Airways"; +

    Note that for all identifiers (bucket names) that contain a hyphen character, you need + to enclose the name with backtick (`) characters.

    -

    - The result is as follows: -

    +

    The results:

    { "requestID": "9e1cd084-f45e-4059-9e7a-edec30f60dd2", @@ -73,60 +57,34 @@ } } -

    - The country is thus specified as United Kingdom. -

    - -
    - -
    - - - Making N1QL queries - -

    - After you install Couchbase Server, you can start using N1QL right away; either with - the interactive query shell, cbq; or with the Query Workbench, - which is provided by the Couchbase Web Console. -

    +

    The country is thus specified as United Kingdom.

    - - Try the Interactive Query Shell - + Try the Interactive Query Shell -

    - To run the interactive query shell, cbq, bring up a console window - on your machine, and enter - the following against the prompt: -

    +

    To run the interactive query shell, cbq, open a console window + on your computer and enter the following:

    - $ bash -c "clear && docker exec -it db sh" + bash -c "clear && docker exec -it db sh" -

    - Then, navigate to the Couchbase bin directory, and start cbq: -

    - - # cd /opt/couchbase/bin -# ./cbq -u Administrator -p password -engine=http://127.0.0.1:8091/ - -

    - This displays the cbq shell prompt, against which you can enter N1QL - commands, specifying your currently installed buckets. - For example, the following query returns - the different values used by documents in the travel-sample bucket - for the callsign - field, limiting the number of results to 5: -

    +

    Then, navigate to the Couchbase bin directory, and start + cbq:

    + + cd /opt/couchbase/bin +./cbq -u Administrator -p password -engine=http://127.0.0.1:8091/ + +

    This displays the cbq shell prompt, against which you can enter N1QL + commands, specifying your currently installed buckets. For example, the following + query returns the different values that are used by the documents in the + travel-sample bucket for the callsign field, + limiting the number of results to five:

    cbq> SELECT callsign FROM `travel-sample` LIMIT 5; -

    - The result is as follows: -

    +

    The results:

    { "requestID": "ae6fcd5b-e7f0-4725-ae1d-a38678c13a3e", @@ -159,19 +117,17 @@ } } -

    - The result thus contains five callsign-values. A callsign is associated with an airline; and - airline is one of the document-types that the travel-sample bucket - contains. Others are airport and hotel. You can search on a type: for example, - the following query returns a maximum of one - airport document, and lists all the fields it contains. -

    +

    The results thus contain five callsign values. A callsign is + associated with an airline; and airline is one of + the document types that the travel-sample bucket contains. Others + are airport and hotel.

    +

    You can also search on a type. For example, the following query returns a maximum of + one airport document, and lists all of the fields that it + contains:

    cbq> SELECT * FROM `travel-sample` WHERE type="airport" LIMIT 1; -

    - The query-result is as follows: -

    +

    The results:

    { "requestID": "c49a5885-9fde-40e3-871f-699f211078cc", @@ -206,13 +162,12 @@ } } -

    - The following query returns the names of (at a maximum) ten hotels that accept pets, - in the city of Medway. -

    +

    The following query returns the names of (at a maximum) ten hotels that accept pets, in the + city of + Medway:cbq> SELECT name FROM `travel-sample` WHERE type="hotel" AND city="Medway" and pets_ok=true LIMIT 10;

    +

    The results:

    - cbq> SELECT name FROM `travel-sample` WHERE type="hotel" AND city="Medway" and pets_ok=true LIMIT 10; -{ + { "requestID": "b6dc75dd-4ed2-40de-83c8-9aebb3820ad8", "signature": { "name": "json" @@ -231,15 +186,15 @@ } } -

    - The - following query returns the name and - phone fields for up to 10 documents for hotels in Manchester, where - directions are not missing; and orders the results by name: -

    +

    The following query returns the name and phone fields for + up to 10 documents for hotels in Manchester, where directions are not missing, and + orders the results by name:

    +

    + cbq> SELECT name,phone FROM `travel-sample` WHERE type="hotel" AND city="Manchester" and directions IS NOT MISSING ORDER BY name LIMIT 10; +

    +

    The results:

    - cbq> SELECT name,phone FROM `travel-sample` WHERE type="hotel" AND city="Manchester" and directions IS NOT MISSING ORDER BY name LIMIT 10; -{ + { "requestID": "a3561cba-2377-4282-9c0f-68fc627950f6", "signature": { "name": "json", @@ -274,124 +229,62 @@
    - - Try the Query Workbench - + Try the Query Workbench -

    - The Couchbase Web Console - provides a Query Workbench, at which you can compose and execute N1QL queries. - Left-click on the Query tab, located on the horizontal control-bar, near - the top of the Couchbase Web Console: -

    - -

    - -

    - -

    - This brings up the Query Workbench. -

    +

    The Couchbase Server Web Console includes the Query Workbench, an interactive tool that lets + you compose and execute N1QL queries. To use the Query Workbench, log into the + Couchbase Server Web Console, and then click Query:

    -

    - The workbench has three principal areas, which are: -

    +

    The Query Workbench has three principal areas:

      -
    • - An upper Query Editor panel, which is - where you will type your N1QL query. - -

      - -

      - -
      • +
    • Query Editor: Where you will type your N1QL query
      • -
    • - A Bucket Insights panel, at the right. This provides information on the - buckets currently maintained by your system. Right now, it shows that just one exists; the - bucket travel-sample. - -

      - -

      - -
      • +
    • Bucket Insights: Provides information on the buckets that are + currently maintained by your system. Right now, it shows that just one exists; + the bucket travel-sample.
      • -
    • - A Query Results panel, at the bottom-left. This shows query-results; and provides - a number of options for their display. To start with, you will use the default option, which is - selectable by the JSON button, and duly displays results in JSON-format. - -

      - -

      - -
      • +
    • Query Results: Shows query results and provides a number of options + for their display. To start with, you will use the default option, which is + selectable by the JSON button, and duly displays results + in JSON-format.
    -

    - You can now use the Query Workbench to enter a N1QL query. In the upper panel, enter the following: -

    +

    Use the Query Workbench to enter the following N1QL query:

    + SELECT name FROM `travel-sample` WHERE callsign = "MILE-AIR";

    -

    - To execute your query, left-click on the Execute button, at the upper-left: -

    +

    To execute your query, click Execute.

    -

    - -

    - -

    - Query-results now appear in the Results panel: -

    +

    The results now appear in the Query Results panel:

    -

    - As you can see, a single document was found to match your specified criterion: which was the document - whose name value is 40-Mile Air (which is, in fact, the document you took an initial - look at, during the previous stage of the Getting Started sequence). -

    +

    As you can see, a single document was found to match your specified criterion — the + document whose name value is 40-Mile Air (which + is, in fact, the document you took an initial look at during the previous step in + this Getting Started sequence).

    - - Next - - - - - - -

    - In the final stage of this Getting Started - sequence, - Choose Your Next Steps - suggestions are provided as to locations you can visit next. -

    +

    The final step in the Getting Started sequence, Choose Your + Next Steps, offers suggestions for learning more about Couchbase and + using it for production use-cases.

    @@ -401,53 +294,38 @@ Other Destinations -

    - In addition to following this brief tutorial, you can learn more about N1QL by looking - at these in-depth resources: -

    -
      -
    • - The online interactive tutorial allows you to learn about N1QL - without having Couchbase Server installed in your own environment. It's a - self-contained tutorial that runs in a web browser and lets you modify the sample - queries. The tutorial covers SELECT statements in detail, including examples of JOIN, - NEST, GROUP BY, and other typical clauses. -
      • +
    • N1QL from + the SDK: Explains how to execute N1QL queries programmatically using + the official Couchbase SDKs.
      • +
    • N1QL Query Language Tutorial: Provides interactive web modules where + you can learn about N1QL without having Couchbase Server installed in your own + environment. The modules are self-contained and let you modify and run sample + queries. The tutorial covers SELECT statements in detail, + including examples of JOIN, NEST, + GROUP BY, and other typical clauses.
      • -
    • - The N1QL cheat sheet provides a concise summary - of the basic syntax elements. Print it out and keep it on your desk where it'll be - handy for quick reference. -
      • +
    • N1QL Cheat Sheet: Provides a concise summary of the + basic syntax elements of N1QL. Print it out and keep it on your desk where it'll + be handy for quick reference.
      • -
    • - The contains - details about N1QL syntax and usage. -
      • +
    • : Describes the N1QL + language structure, including syntax and usage.
      • -
    • - Live and recorded Webinars presented by Couchbase engineers - and product managers highlight features and use cases of Couchbase Server, including - N1QL. Here are some links to webinars devoted entirely to N1QL: Couchbase 103: Querying and Ad hoc Querying for NoSQL. -
      • +
    • Couchbase Webinars: Live and recorded presentations by Couchbase + engineers and product managers that highlight features and use-cases of + Couchbase Server, including N1QL.
      • -
    • - Couchbase - blogs include articles written by Couchbase SDK developers. -
      • +
    • Couchbase + Blog: Regularly-posted technical articles and announcements written + by Couchbase employees, including SDK developers.
      • -
    • The Couchbase forum is a community resource where you can ask questions, find - answers, and discuss N1QL with other developers and the Couchbase team. -
      • +
    • Couchbase + Forum: A community resource where you can ask questions, find + answers, and discuss N1QL with other developers and the Couchbase team.
    diff --git a/content/getting-started/whats-next.dita b/content/getting-started/whats-next.dita new file mode 100644 index 0000000000..ce57a93982 --- /dev/null +++ b/content/getting-started/whats-next.dita @@ -0,0 +1,45 @@ + + + + What's Next? + +

    Ready to learn more about Couchbase Server? Here are some pointers that you can explore:

    +

    +

      +
    • Check out the which walk you through the + steps to create simple applications using Couchbase Server as the backend.
      • +
    • For instructions to install Couchbase Server for development, test, or production + purposes, refer to detailed setup + instructions in the Installation Guide.
      • +
    • + Other links based on persona +
      • +
    +

    +
    + Clean Up Your Demo Container +

    While the ability to quickly spin up and try Couchbase Server using the Docker container + has many benefits, over time you may find yourself running multiple containers with + different base images and different versions of your apps. Eventually, we need to clean up + the demo/test environment before proceeding to install a development or production + environment.

    +

    The following commands help you clean up your demo/test Docker container:

      +
    1. Stop the container using the command: docker stop db
      2. +
    2. Remove the container using the command: docker rm db
      3. +

    +
    +
    Clean Up Your Environment +

    Remember to clean up your existing installation before installing a newer version of Couchbase Server. On Mac OS X, run the following:

      +
    1. Shut down any running instances of Couchbase Server.
      2. +
    2. Delete any previous installations either by using the command line or by dragging the Couchbase Server application to the Trash.
      3. +
    3. Open a Terminal window and delete any application support files from previous + installations: + $ rm -rf ~/Library/Application\Support/Couchbase +$ rm -rf ~/Library/Application\Support/Membase
      4. +
    + +

    +
    + +     diff --git a/content/images/web-console-cluster-overview-buckets.png b/content/images/web-console-cluster-overview-buckets.png index d8f8792d1a..bfe064c74e 100644 Binary files a/content/images/web-console-cluster-overview-buckets.png and b/content/images/web-console-cluster-overview-buckets.png differ diff --git a/content/indexes/indexing-overview.dita b/content/indexes/indexing-overview.dita index 7343436cad..1e672ba0b3 100644 --- a/content/indexes/indexing-overview.dita +++ b/content/indexes/indexing-overview.dita @@ -60,14 +60,12 @@
    Full Text Index
    -
    Full text Index (FTS) enables applications to perform fast keyword searches (Google-like - searches) over their data using the FTS API. Full text index is a global index just like - global secondary indexes. Unlike the view index that indexes only a subset of the data local - to each data node, full text indexes are independently distributed to a set of nodes across - the cluster. Full text indexes can index one or more buckets data. The Full text service is - responsible to handle FTS indexes. Full text Index and Full text service is provided as - a Developer Preview in this version. We do not recommend using the capability in production - systems.
    +
    Full Text Index (FTS) enables applications to perform fast keyword searches (Google-like + searches) over their data using the FTS API. A Full Text Index is a global index just like + Global Secondary Indexes. Unlike the View Index that indexes only a subset of the data local + to each data node, Full Text Indexes are independently distributed to a set of nodes across + the cluster. Full text indexes can index one or more buckets' data. The Full Text Service is + responsible for handling FTS indexes.

    diff --git a/content/indexes/n1ql-in-couchbase.dita b/content/indexes/n1ql-in-couchbase.dita index be117c9e77..a26eee9809 100644 --- a/content/indexes/n1ql-in-couchbase.dita +++ b/content/indexes/n1ql-in-couchbase.dita @@ -24,6 +24,6 @@ possible to create indexes using GSI or view during CREATE INDEX statement with the USING clause.

    - + diff --git a/content/indexes/querying-using-map-reduce-views.dita b/content/indexes/querying-using-map-reduce-views.dita index edaa8e4b03..7eee14c39a 100644 --- a/content/indexes/querying-using-map-reduce-views.dita +++ b/content/indexes/querying-using-map-reduce-views.dita @@ -11,9 +11,9 @@
  • Bucket administrator
  • View administrator
    • For more information on role-based access, see .

    + href="../security/security-roles.dita"/>.

    - + diff --git a/content/indexes/querying-using-spatial-views.dita b/content/indexes/querying-using-spatial-views.dita index 4ee54609f7..e718315080 100644 --- a/content/indexes/querying-using-spatial-views.dita +++ b/content/indexes/querying-using-spatial-views.dita @@ -28,9 +28,9 @@
  • Bucket administrator
  • View administrator
    • For more information on role-based access, see .

    + href="../security/security-roles.dita"/>.

    - + diff --git a/content/indexes/view-indexes-for-n1ql.dita b/content/indexes/view-indexes-for-n1ql.dita index 89ea65c132..317af9ee44 100644 --- a/content/indexes/view-indexes-for-n1ql.dita +++ b/content/indexes/view-indexes-for-n1ql.dita @@ -6,7 +6,10 @@

    Views are deployed into the data service as a local index within the Couchbase Server cluster, aligned with data service distribution. Views provide the following advantages:

    • Auto Partitioning of Indexes: Views come with auto partitioning and smart placement within the data service.
      • -
    • Built-in replicas and HA for Indexing: Views come with built in replicas for high availability within the data service including rack/zone awareness support and rebalance support.
      • +
    • Built-in replicas and HA for Indexing: Views come with built in replicas for high + availability within the data service including Server Group + Awareness support and rebalance support.
    • Simple Scaling Model: Views are automatically placed and scaled with the data service.

    Creating Views for N1QL

    You can define a primary or secondary index using Views for N1QL using the CREATE INDEX statement and USING VIEW clause. For more information on the syntax and examples, see - Deployment Considerations for Virtualized or Containerized Environments - Virtualized platforms such as AWS, VMware, Azure, and Docker containers are popular - ways of achieving hardware scalability to complement Couchbase Server's scalability. In this - section, we describe some considerations when running Couchbase Server on a virtualized - platform. These considerations are common across any virtualized platform or container - infrastructure, and are not specific to Couchbase Server. + Deployment Considerations for Virtual Machines and Containers + Virtualized platforms such as VMware, AWS/Azure/GCP, and Docker (containers) are + popular ways of achieving hardware scalability to complement Couchbase Server's software + scalability. +

    When deploying Couchbase Server on a virtualized platform, some extra considerations should + be made.

    Avoid a Single Point of Failure

    Couchbase Server's resilience and high-availability are achieved through creating a cluster of independent nodes and replicating data between them so that any individual node failure - doesn't lead to loss of access to your data. In a virtualized environment, if you were to run - multiple nodes on the same host or physical hardware, you are inadvertently re-introducing a - single point of failure. In environments where you control the VM placement, we recommend that - each Couchbase Server node runs on a different piece of physical hardware.

    -

    Sizing your Virtualized Platform + doesn't lead to loss of access to your data. In a virtualized environment, if you run multiple + nodes on the same host or physical hardware, you are inadvertently re-introducing a single + point of failure. In environments where you control the virtual machine (VM) placement, it is + recommended that each Couchbase Server node runs on a different piece of physical + hardware.

    +

    Sizing a Virtualized Deployment

    -

    Physical environments tend to provide greater performance, predictability and simplicity over - virtualized environments. Virtualization carries much greater flexibility but does add an - additional layer of complexity which can make both sizing correctly and debugging performance - issues more difficult. We recommend that you have at least 4 cores and, if these are - virtualized, make sure these are dedicated to the VM rather than shared across multiple VMs. - For more information on the hardware requirements, see .

    -

    Disk performance is also an important aspect - and there are many different technologies you - can choose. The general rule of thumb is to make sure that your disk has sufficient throughput - to handle the CRUD operations as well as any indexing, compaction and backup activities that - will be required.

    -

    Avoid Over-committing your Resource

    -

    One of the major benefits of virtualization is the ability to share physical resources and - even over-commit on these resources (meaning each VM can think it has more disk/RAM/CPU - than is actually physically available). One side effect of this is that an individual VM's - performance can vary based on what load other VMs are placing upon the physical resources. If - your application requires consistent performance, ensure that you guard against - over-committing when configuring your VMs. We recommend ensuring your Couchbase Server VMs - have dedicated physical resource and the hardware resource is not over-committed.

    -

    AutoFailover Threshold

    -

    Due to the fact the system is running virtualized on top of a hypervisor or Docker engine, - there will be a minor CPU performance overhead. In certain circumstances this can mean that - our default auto-failover timeout can be a little too sensitive. We recommend increasing the - threshold from 30 to 45 or even 60 seconds. depending how CPU-intensive your workload is.

    +

    The performance characteristics of physical hardware are well understood. Even though VMs + insert a lightweight layer between Couchbase Server and the underlying OS, there is still a + small overhead when running Couchbase Server on a virtual platform.

    +

    For stability and better performance predictability, you should dedicate at least two CPU + cores to a VM in development environments, and four CPU cores to a VM in production.

    +

    Disk performance is also an important factor - and there are many different technologies you + can choose. The general rule of thumb is to make sure that the disk has sufficient throughput + to handle the necessary CRUD operations, as well as any indexing, compaction, and backup + activities that will be required.

    +

    + One of the major benefits of virtualization is the ability to share physical resources + and even over-commit these resources (meaning each virtual instance can think it has + more CPU, RAM, or disk space than is actually physically available). However, in an + over-committed environment, you can end up with containers competing with each other causing + unpredictable performance and sometimes stability issues.

    For Couchbase Server, physical + resources should be dedicated to the VM, rather than shared across multiple VMs. For more + information about hardware requirements, see .

    +

    +

    Auto-Failover Threshold

    +

    Due to the fact that the VM is running on top of a hypervisor or container engine, there will + be a minor CPU performance overhead. In certain circumstances, the default auto-failover + timeout in Couchbase can cause some issues. It is recommended that you change the threshold + from 30 seconds (the default), to 45 or even 60 seconds, depending on how CPU-intensive your + workload is.

    Live Migration

    Some virtualization environments allow VMs to be migrated between physical nodes and/or - between storage backends. When moving the VM there is the potential for small pauses and - network disruption which may affect the scheduler and could trigger a failover. When changing - the backend storage, disk queues, compaction or indexing may be affected, which could have a - knock-on effect on general performance. We recommend that you use Couchbase's built-in - rebalance mechanism for maintenance. If it is absolutely necessary to perform a migration, - then disable auto-failover beforehand and be prepared for a performance impact during the - migration. Pausing / Resuming and snapshotting virtual machines can also have the same effect, - these actions should only be performed on a Couchbase Server node which has been removed from - the cluster.

    -
    Additional Considerations for Containers + between storage backends. When moving the VM, there is the potential for small pauses and + network disruption, which has the potential to affect the scheduler, which in turn could + trigger a failover. When changing the backend storage, disk queues, compaction, or indexing + may be affected, which could have a knock-on effect on general performance. Therefore, it is + recommended that use Couchbase's built-in rebalance mechanism for maintenance.

    +

    If it is absolutely necessary to perform a migration, then disable auto-failover beforehand + and be prepared for a performance impact during the migration. Pausing, resuming, and + snapshotting virtual machines can also have the same effect. These actions should only be + performed on a Couchbase Server node which has been removed from the cluster.

    +
    + Additional Considerations for Containers

    This section lists the additional considerations that are applicable only to containers.

    -

    Map Couchbase Node Specific Data to a Local Folder

    -

    A Couchbase Server Docker container writes all persistent and node-specific data to the directory /opt/couchbase/var by default. It is recommended to map this directory to a directory on the host file system using the -v option to get persistence and performance when using "docker run".

    -

    By mapping the directory /opt/couchbase/var to a directory outside the container with the -v option, you can delete the container and recreate it later without losing your data from Couchbase Server. You can even update to a container running a later release/version of Couchbase Server without losing your data.

    -

    In a standard Docker environment using a union file system, leaving /opt/couchbase/var inside the container results in some amount of performance degradation.

    - If you have SELinux enabled, mounting the host volumes in a container requires an extra - step. Assuming you are mounting the ~/couchbase directory on the host - file system, you need to run the following command once before running your first container - on that host: - mkdir ~/couchbase && chcon -Rt svirt_sandbox_file_t ~/couchbase +

    Map Couchbase Node Specific Data to a Local Folder

    +

    A Couchbase Server container writes all persistent and node-specific data to + /opt/couchbase/var by default. It is recommended to map this + directory to a directory on the host file system using the -v option to get + persistence and performance when using docker run.

    +

    By mapping the directory /opt/couchbase/var to a directory outside the + container with the -v option, you can delete the container and recreate it + later without losing your data from Couchbase Server. You can even update to a container + running a later release/version of Couchbase Server without losing your data.

    +

    In a standard Docker environment using a union file system, leaving + /opt/couchbase/var inside the container results in some amount of + performance degradation.

    + If you have SELinux enabled, mounting the host volumes in a container requires an extra + step. Assuming you're mounting the ~/couchbase directory on the host + file system, you need to run the following command once before running your first container + on that host: + mkdir ~/couchbase && chcon -Rt svirt_sandbox_file_t ~/couchbase

    Increase ULIMIT in Production Deployments

    -

    Couchbase Server normally expects the following changes to ulimits:

    +

    Couchbase Server normally expects the following changes to + ulimits:

    ulimit -n 40960 # nofile: max number of open files ulimit -c unlimited # core: max core file size ulimit -l unlimited # memlock: maximum locked-in-memory address space -

    These ulimit settings are necessary when running under heavy load. If you are just doing light testing and development, you can omit these settings, and everything will still work. To set the ulimits in your container, you need to run Couchbase Docker containers with the following additional --ulimit flags:

    docker run -d --ulimit nofile=40960:40960 +

    These ulimit settings are necessary when running under heavy load. If + you are just doing light testing and development, you can omit these settings, and + everything will still work. To set the ulimits in your container, you need to run Couchbase + Docker containers with the following additional --ulimit flags:

    + docker run -d --ulimit nofile=40960:40960 --ulimit core=100000000:100000000 --ulimit memlock=100000000:100000000 --name db -p 8091-8094:8091-8094 -p 11210:11210 couchbase -

    Since unlimited is not supported as a value, it sets the core and memlock values to 100 GB. If your system has more than 100 GB RAM, increase this value to match the available RAM on the system.

    The --ulimit flags only work on Docker 1.6 or later. +

    Since unlimited is not supported as a value, it sets the + core and memlock values to 100 GB. If your system has + more than 100 GB RAM, increase this value to match the available RAM on the system.

    + The --ulimit flags only work on Docker 1.6 or later.
    diff --git a/content/install/couchbase-using-openshift-container.dita b/content/install/couchbase-using-openshift-container.dita new file mode 100644 index 0000000000..ddc35db6f2 --- /dev/null +++ b/content/install/couchbase-using-openshift-container.dita @@ -0,0 +1,47 @@ + + + + Running Couchbase Server Using Red Hat OpenShift + Using Red Hat OpenShift, it is easy get started with Couchbase Server. This topic walks + you through the steps to get started with the simplest topology: single host, single + container. + +
    + +
    Step 1 Ensure that Red Hat OpenShift Cluster is installed and running in your + environment
    +
    Take a look at the Red Hat OpenShift documentation for details. You + can skip this step if you have Docker Engine installed already.
    +     + +
    Step 2 Download the RHEL 7 image available through the Red Hat Registry
    +
    $docker pull registry.connect.redhat.com/couchbase/server
    +     + +
    Step 3 Start the Couchbase Server container ’db’ using the following command
    +
    $docker run -d --name db -p 8091-8094:8091-8094 -p 11210-11211:11210-11211 couchbase

    This command downloads and runs the container tagged "latest" from the Couchbase repo on the Red + Hat Container Catalog.

    + If you have Couchbase Server running locally on the machine without containers, the + port mappings above under the -p option may fail. Ensure that you stop + your local instance of Couchbase Server before running the above command.
    +     + +
    Step 4 Access the Web Console
    +
    Access the Web Console using http://localhost:8091. If the + container is up and running, you'll see the Couchbase Server Setup Screen.
    +     + +
    Step 5 Set up Couchbase Server
    +
    Walk through the Setup New Cluster wizard and accept the default values. You may need to + lower the RAM allocated to various services to fit within the bounds of the resource for + the containers. For detailed information on setting up the Couchbase server, see . Add the sample buckets ‘travel-sample’ and + ‘beer-sample’ to load sample data that’s ready for immediate experimentation and testing.
    +     +
    +

    You now have a working Couchbase Server instance.

    + +     diff --git a/content/install/deploy-with-docker-hub.dita b/content/install/deploy-with-docker-hub.dita index a4fc8b2bcf..b55b9b6572 100644 --- a/content/install/deploy-with-docker-hub.dita +++ b/content/install/deploy-with-docker-hub.dita @@ -2,7 +2,7 @@ Deployment with Docker Hub - Official Couchbase Server containers on Docker Hub are based on Ubuntu 14.04. + Official Couchbase Server containers on Docker Hub are based on Ubuntu 16.04.
    Container Requirements

    Before you can work with Couchbase Server with Docker containers ensure that Docker Engine is @@ -13,13 +13,64 @@

    For minimum container requirements, follow the minimum hardware recommendations listed here for development, test, and production environments.

    - +

    + +
  • Query Service : No RAM-allocation is required for this service.

    @@ -269,6 +295,16 @@

    • +
  • + Eventing Service : Selection and RAM-allocation for the Eventing Service. + The memory quota + should be 256 MB or more. If this service is selected, a + default quota is provided. +

    + +

    +
    • +

    @@ -283,7 +319,7 @@

  • Index Storage Setting: If the Index Service has been selected, either Standard Global Secondary Indexes or Memory-Optimized Global Secondary Indexes can be chosen here, by means of radio buttons. See - , for details. + , for details.

    @@ -320,7 +356,7 @@ of how to create, edit, flush, and delete buckets can be found in the section Setting Up Buckets. An architectural description of buckets can be found in the section - Buckets. (There + Buckets. (There are three different kinds of bucket, so you may wish to familiarize yourself with their properties, before you start bucket-creation.) Note that sample buckets already contain @@ -338,7 +374,7 @@
    • -
    +
    Join an Existing Cluster @@ -376,14 +412,15 @@ </p> <p> - <image href="../admin/picts/joinWithCustomConfig.png" id="join_cluster_expanded" align="left" width="390"/> + <image href="../admin/picts/joinWithDefaultConfig.png" id="join_with_default_config" align="left" width="390"/> </p> <p> - Newly displayed fields show default values for the four Couchbase services (with each box - checked, indicating that each will be added to this node by default), the + Newly displayed fields show default values for the Couchbase services (a checked + box indicating that the corresponding service will be added to this node), the <uicontrol>Host Name/IP Address</uicontrol> for the current node, and default paths - for <uicontrol>Data</uicontrol> and <uicontrol>Indexes</uicontrol>. Specify + for <uicontrol>Data</uicontrol>, <uicontrol>Indexes</uicontrol>, and + <uicontrol>Analytics</uicontrol>. Specify hostname or IP address, and data and index paths as described above, in <i>Set Up a New Cluster</i>. </p> @@ -395,12 +432,12 @@ the principal cluster-host). However, in some circumstances, node-addition may require addition of a service not previously configured: in which case memory-quota specification is indeed required, and will be prompted for. For example, <i>uncheck</i> the checkboxes - for <uicontrol>Query Service</uicontrol> + for <uicontrol>Query Service</uicontrol>, <uicontrol>Eventing Services</uicontrol>, and <uicontrol>Search Service</uicontrol>. The checkbox-display now appears as follows: </p> <p> - <image href="../admin/picts/joinClusterServiceCheckboxes.png" id="join_cluster_service_checkboxes" align="left" width="220"/> + <image href="../admin/picts/joinClusterServiceCheckboxes.png" id="join_cluster_service_checkboxes" align="left" width="140"/> </p> <p> @@ -519,7 +556,8 @@ Set up your administrator-username and password: <codeblock outputclass="language-bash">curl -v -X POST http://[localhost]:8091/settings/web -d password=[password] - -d username=[admin-name]</codeblock> + -d username=[admin-name] + -d port=[desired-rest-api-port|SAME]</codeblock> <p> Set up the index RAM quota (to be applied across the entire cluster): @@ -563,7 +601,7 @@ curl -u Administrator:password -X POST http://127.0.0.1:8091/pools/default \ </body> <related-links> -<link href="../security/concepts-rba.dita#concept_ntl_jph_hr"/> +<link href="../security/security-roles.dita"/> </related-links> </topic> diff --git a/content/install/install-browsers.dita b/content/install/install-browsers.dita index 1ff563f23f..8fa1d84872 100644 --- a/content/install/install-browsers.dita +++ b/content/install/install-browsers.dita @@ -3,12 +3,12 @@ PUBLIC "-//OASIS//DTD DITA Topic//EN" "topic.dtd"> <topic xml:lang="en-us" id="topic765"> <title>Supported Web Browsers - Couchbase Web Console is supported on Mozilla Firefox, Apple Safari, Google Chrome, - Microsoft Internet Explorer, and Microsoft Edge. + The Couchbase Server Web Console is supported on a variety of modern Web + browsers.

    - Supported Browsers for the Couchbase Web Console + Couchbase Server Web Console Supported Web Browsers @@ -17,41 +17,40 @@ Browser Operating System - Version* + Browser Version Apple Safari - Mac OS X - 10 + macOS + Latest Google Chrome - Mac OS X, Windows - 56 + macOS, Windows + Latest - Mozilla Firefox - Mac OS X, Windows - 52 + Microsoft Edge + Windows + Latest - Internet Explorer + Microsoft Internet Explorer Windows 11 - Microsoft Edge - Windows - 14 + Mozilla Firefox + macOS, Windows + Latest

    -

    * Couchbase Web Console is tested on the listed operating systems (all 64-bit) with the - specified versions of the web browsers.

    + All listed operating systems are 64-bit. diff --git a/content/install/install-client-server.dita b/content/install/install-client-server.dita index 9907665f82..3ac5964b41 100644 --- a/content/install/install-client-server.dita +++ b/content/install/install-client-server.dita @@ -1,7 +1,7 @@ - Client Deployment Strategies Couchbase Server client SDKs are the preferred deployment option. + Couchbase Client Deployment Strategies Couchbase Server client SDKs are the preferred deployment option.

    If your language and development environment supports a smart client library, Couchbase Server client SDKs are the preferred diff --git a/content/install/install-cluster-setup.dita b/content/install/install-cluster-setup.dita index ef0ba5fbdd..784c329b60 100644 --- a/content/install/install-cluster-setup.dita +++ b/content/install/install-cluster-setup.dita @@ -20,7 +20,7 @@

    Multi-Dimensional Scaling (MDS)
    MDS functionality allows separation of Index and Query services from the traditional Couchbase Server data/view services. With MDS and if it fits your use case needs, you diff --git a/content/install/install-intro.dita b/content/install/install-intro.dita index 2c73876ffb..35bda7f78d 100644 --- a/content/install/install-intro.dita +++ b/content/install/install-intro.dita @@ -2,36 +2,58 @@ - Installing Couchbase Server - Couchbase Server installation covers pre-installation, - installation on supported platforms, post-installation, upgrading, cloud deployment and migration procedures. + Installing the Couchbase Server Cluster + Follow this process to install Couchbase Server. -

    Installation involves picking up the platform and the correct software edition for your - needs, downloading the software, and running the installer.

    -

    Before you install Couchbase Server, review the hardware requirements, supported platforms, - network configuration, security considerations, and production deployment considerations. Then - follow the installation instructions for your operating system or cloud environment.

    - -
    - Download Couchbase Server -

    Couchbase Server is supported on numerous popular operating systems. Look for the binaries - for your operating system on the Couchbase download page.

    -

    Each of the binaries listed on the download page has appropriate installation - instructions.

    -

    For example, when selecting installation instructions next to the Ubuntu package, the following is shown: -

    -
      -
    1. Download or transfer the download to your Ubuntu system.
      2. - -
    2. Install the package using the dpkg command as a privileged user under - sudo. For example: - sudo dpkg -i couchbase-server-enterprise_5.0.0-ubuntu14.04_amd64.deb
      3. - - -
    -
    + + Installing Couchbase Server + + + + + + + Step + Action + Description + + + + + Step 1 + Review the system requirements and deployment guidelines + Couchbase Server has a set of system requirements and deployment guidelines that + vary depending on your deployment. Make sure that you plan your environment + accordingly before installation.

     +     + + Step 2 + Install Couchbase Server on each node in the cluster + Couchbase is a clustered database and requires that Couchbase Server is installed + and running on each node before joining them together into a cluster.

     +     + + Step 3 + Verify the installation and node availability + After installing Couchbase Server on a node, you can do a basic verification of + the installation by confirming access to the Couchbase Server Web Console. You can + also do other advanced verifications, depending on your needs.

     +     + + Step 4 + Initialize the Couchbase Server cluster + After you install Couchbase Server on all of the nodes, you need to join them + together into a cluster.

     +     + + +
    diff --git a/content/install/install-linux.dita b/content/install/install-linux.dita index 7022b28841..2359ad9248 100644 --- a/content/install/install-linux.dita +++ b/content/install/install-linux.dita @@ -7,13 +7,22 @@

    You can download Couchbase Server for your OS directly from .

    -

    Links to the distribution-specific installation instructions for Linux can be found below:

      Links to the distribution-specific installation instructions for Linux can be found below: +

      + +
      • Red Hat Enterprise Linux and CentOS
      • Ubuntu and Debian
      • SUSE Enterprise Linux
      • Oracle Enterprise Linux
        • -

      +
    + +

    + For information on installing Couchbase Server on IPv6 systems, see + IP Version 6: Set-Up Procedures. +

    + RHEL6 and other newer Linux distributions are known to have Transparent Huge Pages (THP) feature enabled by default. THP can cause issues in database software, including Couchbase Server. THP can cause nodes to lose cluster heartbeats, resulting in erroneous diff --git a/content/install/install-package-windows.dita b/content/install/install-package-windows.dita index 3128fcfec3..1b18ce7617 100644 --- a/content/install/install-package-windows.dita +++ b/content/install/install-package-windows.dita @@ -42,6 +42,12 @@ ensure the universal CRT is installed. For other versions of Windows, see Update for Universal C Runtime in Windows.

    + +

    + For information on installing Couchbase Server on IPv6 systems, see + IP Version 6: Set-Up Procedures. +

    +
    diff --git a/content/install/install-platforms.dita b/content/install/install-platforms.dita index 94f3e52f17..b9fa6662cc 100644 --- a/content/install/install-platforms.dita +++ b/content/install/install-platforms.dita @@ -2,136 +2,105 @@ - Supported Platforms + Supported Operating Systems Couchbase Server is supported on numerous popular operating systems and virtual - environments. All options listed below are 64-bit compatible. + environments. - Supported Platforms for Development, Testing, and Production +

    Certified: Couchbase has performed an extensive testing and review process to make + sure that Couchbase Server runs well out of the box on the specified platform.

    +

    Supported: Couchbase believes that Couchbase Server will run well out of the box + on the specified platform, but extensive testing has not been performed. Couchbase will + make a best effort to investigate issues on a Supported platform version, but it is + recommended that you run Couchbase on a Certified platform version.

    + All listed operating systems are 64-bit. +
    + Supported Operating Systems for Development, Testing, and Production - - + + - Platform + Operating System Version - Red Hat Enterprise Linux - Certified 7.2, Supported 7.x + Amazon Linux* + Certified 2017.09 - Red Hat Enterprise Linux - Certified 6.5, Supported 6.x - - - SUSE Enterprise Linux - Certified 12 - - - SUSE Enterprise Linux - Certified 11.4, Supported 11.x - - - Oracle Enterprise Linux* - Certified 7 - - - Oracle Enterprise Linux* - Certified 6.6 - - - Ubuntu Linux - Certified 16.04 + CentOS + Certified 7.2, 7.4; Supported 7.x

    Certified 6.7, 6.9; Supported + 6.x

         - Ubuntu Linux - Certified 14.04 + Debian +

    Certified 9; Supported 9.x

    Certified 8         - CentOS - Certified 7.2, Supported 7.x - - - CentOS - Certified 6.5, 6.7, Supported 6.x + Oracle Linux** + Certified 7.3

    Certified 6.6

         - Debian Linux - Certified 8 + Red Hat Enterprise Linux (RHEL) + Certified 7.2; Supported 7.x

    Certified 6.5; Supported + 6.x

         - Debian Linux - Certified 7 + SUSE Linux Enterprise Server (SLES) + Certified 12; Supported 12.x

    Certified 11.4; Supported + 11.x

         - Amazon Linux** - 2017.09 + Ubuntu + Certified 16.04

    Certified 14.04

         - Windows 2016 + Windows Server -

    Certified Standard, Supported with monthly roll-up updates

    +

    Certified 2016 Standard; Supported with monthly rollups

    +

    Certified 2012 R2 Standard; Supported with monthly rollups

         - - Windows 2012 - Certified Standard R2, Supported with monthly roll-up updates - -

    Supported Platforms for Development and Testing - +
    +

    * Amazon Linux is only supported when it is run using the official Couchbase Server + AMI on AWS.

    +

    ** Only the Red Hat Compatible Kernel (RHCK) is supported. The Unbreakable Enterprise + Kernel (UEK) is not supported.

    +

    + + Supported Operating Systems for Development and Testing + - + - Platform + Operating System Version - Windows 10 - Anniversary Edition - - - Mac OS X - Certified 10.12 (Sierra) + macOS + Certified 10.11 (El Capitan), 10.12 (Sierra), 10.13 (High + Sierra) - Mac OS X - Certified 10.11 + Windows + Certified 10 Anniversary Update -

    -

    Certified: Certified means that on this OS version, Couchbase Server has passed our - extensive testing and review process to make sure it runs well out of the box.

    -

    Supported: Supported means that on this OS version, we believe that Couchbase Server will - run well out of the box but extensive testing has not been carried out with that OS - version. We will make our best effort to investigate the issue on a compatible OS - version, but recommend you run Couchbase on the certified OS version.

    -

    * Only the Red Hat Compatible Kernel (RHCK) is supported, the Unbreakable Enterprise Kernel - is not a supported platform.

    -

    ** Amazon Linux only supported when run using the official Couchbase Server AMI on - AWS.

    -

    The following platforms are no longer supported:

    -

      -
    • Microsoft Windows 7
      • -
    • Microsoft Windows 8
      • -
    • Microsoft Windows 2008
      • -
    • Amazon Linux 2014.03, 2016.03
      • -
    • Ubuntu Linux 12.04
      • -
    • Mac OS X 10.10, 10.9, 10.8
      • -

    For more information on the supported web browsers, see .

    -

    For more information on deprecated items, see Release Notes.

    - - - + +

    +

    + For information about the platform support changes in this release, see the release + notes.

    For information about supported web browsers, see .

    +

    -     diff --git a/content/install/install-ports.dita b/content/install/install-ports.dita index 01d7807a54..0e002187d4 100644 --- a/content/install/install-ports.dita +++ b/content/install/install-ports.dita @@ -2,393 +2,460 @@ - Network Configuration - Couchbase Server uses specific network ports for communication between server components - and with the clients accessing the data stored in the Couchbase Server cluster. + Network and Firewall Requirements + Couchbase Server uses a variety of network ports for communication between server + components and with the clients that access the data stored in the Couchbase Server + cluster. -

    The ports listed in the following table must be open on the host for Couchbase Server to operate correctly. - Couchbase Server configures these ports automatically, but you must verify that any - firewall configuration allows communication on the specified ports for each usage type. - If this is not done, the node can experience difficulty in joining a cluster, - including acting as a cluster of one node rather than joining the intended cluster.

    +

    The ports listed below must be open on each host for Couchbase Server to operate correctly. In + addition, certain ports must be available (i.e., not blocked by a firewall or other such + mechanism) between each node of a cluster, between nodes of multiple clusters connected + via XDCR, between application servers and nodes, and for administrative access.

    + If any port numbers are already in use by other running applications, + Couchbase Server will not function properly and may fail to start. -

    The following table lists the ports used for different types of Couchbase Server network communication:

    +

    The following is a list of port numbers grouped by category of communication path:

    -
    Node-to-node
    -
    Couchbase Server uses these ports for communication between all nodes within the cluster. - These ports must be open to enable nodes to communicate with each other.
    +
    Node-local: Only connected to over localhost, needs to be open on the node + but not available externally.
    +
    11213, 9119, 9998
     - - -
    Node-to-client
    -
    These ports should be open between each node within the cluster and any - client nodes accessing data within the cluster.
    -     -
    -
    - -
    Cluster administration
    -
    Couchbase Server uses these ports for administration via the REST API, command-line clients, and web browsers.
    -     - -
    XDCR
    -
    These ports are used for XDCR (Cross Datacenter Replication) communication between all nodes - in both the source and destination clusters.
    -     -
    - - - - - - - - - - - - - - - Port - Description - Node-to-node - Node-to-client* - Cluster administration - XDCR v1 (CAPI) - XDCR v2 (XMEM) - - - - - 8091 - Used by the Couchbase Web Console for REST/HTTP traffic. - Yes - Yes - Yes - Yes - Yes - - - 8092 - Used to access views, run queries, XDCR and update design documents. - Yes - Yes - No - Yes - Yes - - - - 8093 - Used by query services for REST/HTTP traffic. - Yes - Yes - No - Yes - Yes - - - 8094** - External HTTP port for the Search service - Yes - Yes - Yes - No - No - - - - - 9100 - Internal index admin port (-adminPort). - Yes - No - No - No - No - - - 9101 - Internal index scan port (-scanPort). - Yes - No - No - No - No - - - 9102 - Internal index HTTP port (-httpPort). - Yes - No - Yes - No - No - - - 9103 - Internal index build port (-streamInitPort). - Yes - No - No - No - No - - - 9104 - Internal index build port (-streamCatchupPort ). - Yes - No - No - No - No - - - 9105 - Internal index maintenance port (-streamMaintPort). - Yes - No - No - No - No - - - 9998 - Internal REST port. - No - No - Yes - Yes - Yes - - - 9999 - Internal GSI port used by the internal administrators. - Yes - No - No - No - No - - - - 11207*** - Used by smart client libraries to access data nodes using SSL. This is a memcached SSL - port. - No - Yes - No - No - Yes when SSL is used - - - 11209 - Internal Bucket Port. - Yes - No - No - No - No - - - 11210*** - Used by smart client libraries or Moxi to directly connect to the data nodes. The XDCR - client uses this port as well as the SDKs. This is a memcached - port. - Yes - Yes - No - No - Yes - - - 11211 - Used by pre-existing Couchbase Server and memcached (non-smart) client libraries (such - as Moxi). - No - Yes - No - No - No - - - 11214 - Used for SSL XDCR data encryption. - No - No - No - No - Yes - - - 11215 - Used for SSL XDCR data encryption. - No - No - No - No - Yes - - - 18091 - Used by the Couchbase Web Console for REST/HTTP traffic with SSL. - No - Yes - Yes - No - Yes - - - 18092 - Used to access views, run queries, XDCR and update design documents with SSL. - No - Yes - No - No - Yes - - - 18093 - N1QL SSL port - Yes - Yes - No - Yes - Yes - - - 18094** - External HTTPS port for the Search service - Yes - Yes - Yes - No - No - - - 4369 - Erlang Port Mapper ( epmd ) - Yes - No - No - No - No - - - 21100 to 21299 (inclusive) - Node data exchange. - Yes - No - No - No - No - - - -
    -
      -
    • * bi-directional
      • -
    • ** This port was introduced in Couchbase Server 4.5.
      • -
    • *** This port has to be open for XDCR to connect to target memcached.Ports - 6060 and 11213 are used internally on the local host. They are not used for - communication between cluster nodes. For firewall purposes, you do not need to - take these ports into consideration. However, if another service is listening on - either of these ports, the Couchbase Server service will not start.
      • - -
    -

    In summary, for the following, respective tasks, open these listed ports in your firewall.

    -
    -
    Node-to-node
    -
    For communication between all nodes within the cluster, you won't normally be using a firewall but these are the ports used:
    -
    4369, 8091, 8092, 8093, 8094 (Couchbase Server 4.5 and upwards), 9100, 9101, 9102, 9103, 9104, 9105, 9999, 11209, 11210, 18093, 21100-21299 (inclusive).
    -     - -
    Node-to-client
    -
    8091, 8092, 8093, 8094 (Couchbase Server 4.5 and upwards), 11207, 11210, 11211, 18091, 18092, 18093.
    +
    Node-to-node: Between all nodes within a single cluster.
    +
    4369, 8091-8096, 9100-9105, 9110-9118, 9120-9122, 9999, 11209, 11210, 21100-21299
     -
    Cluster administration
    -
    8091, 8094 (Couchbase Server 4.5 and upwards), 9102, 9998, 18091.
    +
    Client-to-node: Between any clients/app-servers/SDKs and + all nodes of each cluster they require access to.
    +
    Unencrypted*: 8091-8096, 11210, 11211
    +
    Encrypted: 18091-18096†††, 11207
     -
    XDCR v1 (CAPI)
    -
    8091, 8092, 8093, 9998, 18093.
    +
    Cluster administration: Administration traffic via the REST API, + command-line, and Web UI.
    +
    Unencrypted*: 8091
    +
    Encrypted: 18091
    +
    Note: certain support/diagnostic requests may run against + ports other than 8091. These are expected to execute locally on a node and + so do not require external access.
     -
    XDCR v2 (XMEM)
    -
    8091, 8092, 8093, 9998, 11207 (when SSL used), 11210, 11214, 11215, 18091, 18092, 18093.
    +
    XDCR: Between all source and destination nodes of an XDCR replication stream.
    +
    v1 (CAPI)
    +
    8091, 8092
    +
    v2 (XMEM)
    +
    Unencrypted*: 8091, 8092, 11210
    +
    Encrypted: 8091, 11207, 18091, 18092
    -
    User-Defined Ports -

    You can install and run Couchbase Server with user-defined ports rather than with the default ports.

    - If you want to run Couchbase Server on user-defined ports and with multiple - instances running on a single machine, you have to keep in mind that the standard RPM - installation for Linux doesn’t allow you to specify separate installation paths needed - for separate multiple instances. Therefore, you must perform multiple non-root (sudo or - non-sudo) installations instead. -

    Setting up Couchbase Server with User-Defined Ports

    +

    * If enforcing encryption (SSL/TLS), these ports may be blocked + outside of a Couchbase Server cluster but need to remain open between nodes.

    + +

    The following table provides more description:

    +

    + + + + + + + + + + + + + Port Name + Default Port Number Un / Encrypted + Description + Node-to-node + Client-to-Node + Cluster admin + XDCR v1 (CAPI) + XDCR v2 (XMEM) + + + + + epmd port† + 4369 + Erlang Port Mapper Daemon + Yes + No + No + No + No + + + rest_port / ssl_rest_port + 8091 / 18091 + REST/HTTP including Web UI + Yes + Yes + Yes + Yes + Yes + + + capi_port / ssl_capi_port + 8092 / 18092 + Views and XDCR access + Yes + Yes + No + Yes + Yes + + + query_port / ssl_query_port + 8093 / 18093 + Query service REST/HTTP traffic + Yes + Yes + No + No + No + + + fts_http_port / fts_ssl_port + 8094 / 18094 + Search service REST/HTTP traffic + Yes + Yes + No + No + No + + + cbas_http_port / cbas_ssl_port + 8095 / 18095††† + Analytics service REST/HTTP traffic + No + Yes + No + No + No + + + eventing_http_port/ eventing_ssl_port + 8096 / 18096 + Eventing service REST/HTTP traffic + No + Yes + No + No + No + + + debugPort††††† + 9140 + Port for Eventing Debugger + No + Yes + No + No + No + + + indexer_admin_port + 9100 + Indexer service + Yes + No + No + No + No + + + indexer_scan_port + 9101 + Indexer service + Yes + No + No + No + No + + + indexer_http_port + 9102 + Indexer service + Yes + No + No + No + No + + + indexer_stinit_port + 9103 + Indexer service + Yes + No + No + No + No + + + indexer_stcatchup_port + 9104 + Indexer service + Yes + No + No + No + No + + + indexer_stmaint_port + 9105 + Indexer service + Yes + No + No + No + No + + + cbas_admin_port + 9110 + Analytics service + Yes + No + No + No + No + + + cbas_cc_http_port + 9111 + Analytics service + Yes + No + No + No + No + + + cbas_cc_cluster_port + 9112 + Analytics service + Yes + No + No + No + No + + + cbas_cc_client_port + 9113 + Analytics service + Yes + No + No + No + No + + + cbas_console_port + 9114 + Analytics service + Yes + No + No + No + No + + + cbas_cluster_port†††† + 9115 + Analytics service + Yes + No + No + No + No + + + cbas_data_port + 9116 + Analytics service + Yes + No + No + No + No + + + cbas_result_port + 9117 + Analytics service + Yes + No + No + No + No + + + cbas_messaging_port + 9118 + Analytics service + Yes + No + No + No + No + + + cbas_auth_port + 9119 + Analytics service + No + No + No + No + No + + + cbas_replication_port + 9120 + Analytics service + Yes + No + No + No + No + + + cbas_metadata_port + 9121 + Analytics service + Yes + No + No + No + No + + + cbas_metadata_callback_port + 9122 + Analytics service + Yes + No + No + No + No + + + xdcr_rest_port + 9998 + XDCR REST port + No + No + No + No + No + + + projector_port + 9999 + Indexer service + Yes + No + No + No + No + + + memcached_dedicated_port + 11209 + Data Service + Yes + No + No + No + No + + + memcached_port / memcached_ssl_port + 11210 / 11207 + Data Service + Yes + Yes + No + No + Yes + + + moxi_port†† + 11211 + Moxi port + No + Yes + No + No + No + + + moxi_port_internal†† + 11213 + Moxi port + No + No + No + No + No + + + Internal data ports + 21100 to 21299 (inclusive) + Node data exchange. + Yes + No + No + No + No + + + +
    +

    +

    † Cannot be remapped

    +

    †† Cannot be remapped. Deprecated in 5.0 and will be removed in a subsequent release. Consider + using client-side Moxi instead.

    +

    ††† Analytics SSL port not currently used; reserved for future use.

    +

    †††† In 5.5 Beta build, this port was named cbas_hyracks_console_port.

    +

    ††††† The Eventing debugger port, debugPort, is an internal port and is not supported for + external access outside of the cluster. Ensure to use this port only in your developer + environments.

    + +
    Custom Port Mapping + Changing the port mappings will require a reset and + reconfiguration of any Couchbase Server node. + +

    Setting up Couchbase Server with Custom Ports

      -
    1. Install Couchbase Server -
        -
      • If Couchbase Server is already installed and running, stop it.
        • -
      2. -
    2. Add the new user-defined ports to the /opt/couchbase/etc/couchbase/static_config - file.
        -
      • The /opt/couchbase/etc/couchbase/static_config is the location - from where the Couchbase Server picks up the configuration parameters.
        • -
      • If port numbers are not specified, default ports are used.
        • -
      • To override some or all default ports, append the user-defined ports to the file.
        • -
      3. +
    3. Install Couchbase Server (If already running, stop it.)
      4. +
    4. Add the new user-defined ports to the + /opt/couchbase/etc/couchbase/static_config file (this + will be at wherever you put <path to> + /couchbase/etc/couchbase/static_config for multiple node + installations).
        +
      • For example, to change the REST API port from 8091 to 9000, add this + line: {rest_port, 9000}
        • +
      • Any ports not listed will be assigned their defaults as listed + above
        • +
    5. (Optional) CAPI port (default 8092) can be edited in the - /opt/couchbase/etc/couchdb/default.d/capi.ini file by replacing 8092 with the new - port name.
      6. -
    6. If the Couchbase Server was previously configured, delete the - opt/couchbase/var/lib/couchbase/config/config.dat file to remove - the old configuration.
      7. + /opt/couchbase/etc/couchdb/default.d/capi.ini file + by replacing 8092 with the new port number. +
    7. If Couchbase Server was previously configured, delete the + opt/couchbase/var/lib/couchbase/config/config.dat file + to remove the old configuration.
    8. Start Couchbase Server.
    - -

    Ports to Change

    -

    The following are the user-defined ports to add, replace, or append to the - /opt/couchbase/etc/couchbase/static_config file.

    - {rest_port, 9000}. -{mccouch_port, 8999}. -{memcached_port, 12000}. -{query_port, 9409}. -{ssl_query_port, 19409}. -{fts_http_port, 9201}. -{moxi_port, 12001}. -{short_name, "ns_1"}. -{ssl_rest_port,11000}. -{ssl_capi_port,11001}. -{ssl_proxy_downstream_port,11002}. -{ssl_proxy_upstream_port,11003}. - - If the newly configured ports overlap with ports used by other running - applications, Couchbase Server fails to start. If the newly configured ports overlap with - ports used by Couchbase buckets, Erlang crash notifications appear in the log file. - In order to set up multiple nodes per machine, you need to assign unique - values to these ports. - -

    How to Map User-Defined Ports to the Default Ports

    -

    This chart shows how certain user-defined ports are mapped to the default ports: {rest_port, 9000} 8091 Web administration port -{mccouch_port, 8999} 11213 Default value for mccouch -{memcached_port, 12000} 11211 Client interface (proxy) -{query_port, 9409} 8093 Default query port over HTTP -{ssl_query_port, 19409} 18093 Default query port over SSL -{fts_http_port, 9201} 8094 Default FTS port -{moxi_port, 12001} 11210 Internal/external bucket port -{ssl_rest_port,11000} 18091 Internal REST HTTPS for SSL -{ssl_capi_port,11001} 18092 Internal CAPI HTTPS for SSL -{ssl_proxy_downstream_port,11002} 11214 Incoming SSL proxy -{ssl_proxy_upstream_port,11003} 11215 Internal outgoing SSL proxy - All default ports are reserved and cannot be used for other - purposes. -

    diff --git a/content/install/install-production-deployment.dita b/content/install/install-production-deployment.dita index 1084185366..9d4140bb37 100644 --- a/content/install/install-production-deployment.dita +++ b/content/install/install-production-deployment.dita @@ -1,41 +1,116 @@ - Sizing and Other Deployment Considerations - To configure the Couchbase Server production environment, follow the recommendations - about the cluster setup, sizing guidelines, swap space, and client server deployment - strategies. + Deployment Guidelines + Before you install Couchbase Server, follow the recommended deployment guidelines for + setting up your production environment. - - -

    Proper configuration of I/O subsystems is critical for the optimal performance and - operation and performance of Couchbase Server. These are the most common best practices recommended for storage configuration:

    - -
    General Recommendations - - + + Couchbase Server Deployment Guidelines + + + + + + Guideline + Description + + + + + Sizing + Evaluate the overall performance and capacity requirements of your + deployment, and determine the hardware and other resources.

     +     + + Time keeping + Keeping accurate time is essential to a properly-functioning + database. Ensure that you follow the guidelines for synchronizing each + Couchbase node using Network Time Protocol (NTP).

     +     + + Disable Transparent Huge Pages + You must disable the THP memory management system on each node that + runs Couchbase Server.

     +     + + Set kernel swappiness + The kernel swappiness setting defines how aggressively + the kernel will swap memory pages versus dropping pages from the page + cache. You need to set the swappiness setting to 0, or at most 1, for + optimal Couchbase Server operation.

    +     + + Couchbase client deployment + When you deploy Couchbase Server, you need to plan how clients will + access and interact with the cluster.

     +     + + Security + Couchbase Server provides security features that allow administrators + to implement various security controls to ensure secure deployment. You + should take Couchbase security best practices into consideration before, + during, and after deployment.

     +     + + Virtualized and containerized deployments + Certain considerations must be made when you're deploying Couchbase + Server on a virtual machine or container.

     +     + + Clusters with less than three nodes + Couchbase Server clusters with less than three nodes are not + recommended in production. However, you might find a need to have a + smaller deployment for test and development purposes.

     +     + + +
    +
    + General Guidelines
    Use RAID 1+0 (or RAID 1) if Couchbase bucket replication is 1 or lower
    -
    If bucket replicas are set to 2 or higher, the administrator might rely on database-level - replication for data durability.
    +
    If bucket replicas are set to 2 or higher, the administrator might rely on + database-level replication for data durability.
     -
    +
    +
    -
    Multi-Dimensional Scaling (MDS)
    -
    MDS functionality allows separation of Index and Query services from the traditional - Couchbase Server data/view services. With MDS and if it fits your use case - needs, you can have different disk storage configurations for cluster nodes - running the Data service and different ones running the Index service. For - example, nodes running the Index service could have SSD backed storage for - better performance. Refer to for more details.
    +
    Linux OS
    +
    +
      +
    • When deploying Couchbase Server on production Linux, you should use + either the XFS or ext4 file system. XFS is preferred because it + provides better performance when multiple IO streams are accessing + the file system, such as during Data Service compaction or when the + working set exceeds the memory quota and the data is read from + disk.
      • +
    • The Couchbase Linux installer requires the ability to create a + normal local Unix user with the name couchbase, if + such a user doesn't already exist. This can be particularly + important if you are using a directory service for user management, + such as LDAP.
      • +
    +
    -
    +
    Windows OS
    @@ -43,28 +118,25 @@
    - -
    Nodes Running the Data Service - +
    + Nodes Running the Data Service
    -
    Isolate the data and indexes
    -
    For the best performance, isolate the data and indexes (note that these are different from - the Index Service) at the physical disk level.
    - +
    Isolate the data and indexes
    +
    For the best performance, isolate the data and indexes (note that these + are different from the Index Service) at the physical disk level.
    Combine different workloads
    -
    Combining different workloads (workloads with very different I/O and latency characteristics) - can have a negative effect on the overall performance. In case of combined workloads, it might - be appropriate to have objects in different buckets and the files for each bucket on different - mount points/paths.
    +
    Combining different workloads (workloads with very different I/O and + latency characteristics) can have a negative effect on the overall + performance. In case of combined workloads, it might be appropriate to + have objects in different buckets and the files for each bucket on + different mount points/paths.
    - -
    diff --git a/content/install/install-security-bp.dita b/content/install/install-security-bp.dita index 8f03bd74f1..d697fbd531 100644 --- a/content/install/install-security-bp.dita +++ b/content/install/install-security-bp.dita @@ -2,24 +2,14 @@ Security Considerations - By following security best practices, you can mitigate potential security issues + Ensure that you follow security best best practices throughout the deployment + lifecycle. -

    Security is important for every line of business. This section details some security best - practices that you should consider both before and after you install Couchbase Server. See - also for more - details.

    -

    While you are setting up your server environment, be sure to do the following:

    - -
    - -
    Enhance physical security
    -
    To enhance physical security of your Couchbase Server clusters, consider the following points:
      -
    • Ensure that only authorized personnel have physical access to the Couchbase Server cluster environment.
      • - -
    • Regularly back up all data and secure backups in an off-site location.
      • -
    -     -
    +

    Security is important for every line of business. This topic summarizes some security best + practices that you should consider both before and after you install Couchbase Server. You + should also review the other topics under before deploying Couchbase.

    +

    While you are setting up your server environment, be sure to do the following:

    @@ -65,8 +55,7 @@
    Secure the file system
    You might also consider encrypting your file system to ensure that only authorized processes and users can access it. Many third-party libraries are available for - transparently encrypting Couchbase Server data without any application changes. See for more information.
    + transparently encrypting Couchbase Server data without any application changes.
    @@ -80,6 +69,17 @@
    +
    + +
    Enhance physical security
    +
    To enhance physical security of your Couchbase Server clusters, consider the following + points:
      +
    • Ensure that only authorized personnel have physical access to the Couchbase Server + cluster environment.
      • +
    • Regularly back up all data and secure backups in an off-site location.
      • +
    +     +
    diff --git a/content/install/installation.ditamap b/content/install/installation.ditamap index 9c7e6b221f..4bc9c5ac0e 100644 --- a/content/install/installation.ditamap +++ b/content/install/installation.ditamap @@ -5,86 +5,88 @@ - - - - - - - - - + + + + + + + + - - + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + - - - - - - + + + + + + - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/install/ipv6-setup.dita b/content/install/ipv6-setup.dita new file mode 100644 index 0000000000..5a1874c8ba --- /dev/null +++ b/content/install/ipv6-setup.dita @@ -0,0 +1,89 @@ + + + + + + +Using IPv6 With Couchbase + + Couchbase Server supports IP version 6 (IPv6). Set-up is required for each platform. + + + IPv6 support is only available in Couchbase Server Enterprise + Edition. + +
    + IPv6 Overview +

    By default, when Couchbase Server starts up it uses IPv4. To enable IPv6, set-up is + required for each platform, as described in the sections below. Note that the steps for + Linux and macOS are to be performed after the basic Couchbase Server installation is + complete, and the steps for Windows are integrated into the basic installation procedure + itself.In a clustered setup, IPv6 is enabled only when IPv6 is enabled on all of the + nodes.

    +

    See for more information about + installation procedures.

    +
    + +
    + + IPv6 Set-Up for Linux + +

    After the Couchbase Server package has been installed, the couchbase service + is started by default. To enable IPv6, proceed as follows:

    + +
      + +
    1. Stop the couchbase-server service.
      2. + +
    2. In /opt/couchbase/etc/couchbase/static_config, set ipv6 to + true.
      3. + +
    3. Delete /opt/couchbase/var/lib/couchbase/config/config.dat.
      4. + +
    4. Restart the couchbase-server service (this starts Couchbase Server in IPv6 + mode). To check, point a supported web browser to http://[::1]:8091, + which is the IPv6 address and port number on which Couchbase Server should be + running.
      5. +
    +
    + +
    + + IPv6 Set-Up for Windows + +

    IPv6 set-up requirements are incorporated into the InstallShield wizard itself. During + installation of the Couchbase package, a check box is available to enable IPv6 mode. (Note + that after installation, any subsequent mode-change requires re-installation.)

    + +
    + +
    + + IPv6 Set-Up for macOS + +

    After you install the Couchbase Server application, proceed as follows:

    + +
      +
    1. Ensure that the couchbase-server service is stopped. (Quit the Couchbase + Server application if it is running.)
      2. + +
    2. In /Applications/Couchbase\ + Server.app/Contents/Resources/couchbase-core/etc/couchbase/static_config.in, + set ipv6 to true.
      3. + +
    3. Delete ~/Library/Application\ + Support/Couchbase/var/lib/couchbase/config/config.dat.
      4. + +
    4. Restart the couchbase-server service (this starts Couchbase Server in IPv6 + mode). To check, point a supported web browser to http://[::1]:8091, + which is the IPv6 address and port number on which Couchbase Server should be + running.
      5. + +
    + +
    + + + +     diff --git a/content/install/macos-install.dita b/content/install/macos-install.dita index d2dad21a3f..40005317bc 100644 --- a/content/install/macos-install.dita +++ b/content/install/macos-install.dita @@ -18,6 +18,11 @@ Couchbase Server distribution. It is more difficult to diagnose non-functioning or damaged installations after extraction by other third party archive extraction tools.

    +

    + For information on installing Couchbase Server on IPv6 systems, see + IP Version 6: Set-Up Procedures. +

    +
    Prerequisites

    Delete any previous installations of Couchbase Server using the command line or by dragging the icon to the Trash can.

    @@ -66,7 +71,7 @@
  • Unzip the package containing Couchbase Server: open couchbase-server-version.zip
  • Move Couchbase Server application to your /Applications folder: - mv couchbase-server-version/Couchbase\ Server.app/Applications/
    • + mv couchbase-server-version/Couchbase\ Server.app /Applications/
  • Start Couchbase Server from the terminal:open /Applications/Couchbase\ Server.app This enables you to use Couchbase Server as a non-sudo, non-root user.
    • diff --git a/content/install/picts/addServerNode.png b/content/install/picts/addServerNode.png index 99450b38b0..a50e649d25 100644 Binary files a/content/install/picts/addServerNode.png and b/content/install/picts/addServerNode.png differ diff --git a/content/install/picts/addserver.png.png b/content/install/picts/addserver.png.png new file mode 100644 index 0000000000..939a16b017 Binary files /dev/null and b/content/install/picts/addserver.png.png differ diff --git a/content/install/picts/cb-initial-setup-5.0.1.png b/content/install/picts/cb-initial-setup-5.0.1.png new file mode 100644 index 0000000000..872ced0bee Binary files /dev/null and b/content/install/picts/cb-initial-setup-5.0.1.png differ diff --git a/content/install/picts/cluster-setup-add-server-db1.png b/content/install/picts/cluster-setup-add-server-db1.png new file mode 100644 index 0000000000..c5196a66f1 Binary files /dev/null and b/content/install/picts/cluster-setup-add-server-db1.png differ diff --git a/content/install/picts/docker-multi-machine-db123.png b/content/install/picts/docker-multi-machine-db123.png new file mode 100644 index 0000000000..1f2bd8566d Binary files /dev/null and b/content/install/picts/docker-multi-machine-db123.png differ diff --git a/content/install/picts/docker-single-machine-db123.png b/content/install/picts/docker-single-machine-db123.png new file mode 100644 index 0000000000..1f2bd8566d Binary files /dev/null and b/content/install/picts/docker-single-machine-db123.png differ diff --git a/content/install/picts/macosx-menubar.png b/content/install/picts/macosx-menubar.png index 0c733e10d5..4876ab86cd 100644 Binary files a/content/install/picts/macosx-menubar.png and b/content/install/picts/macosx-menubar.png differ diff --git a/content/install/picts/rebalance.png b/content/install/picts/rebalance.png new file mode 100644 index 0000000000..e2d8fc9a91 Binary files /dev/null and b/content/install/picts/rebalance.png differ diff --git a/content/install/plan-for-production.dita b/content/install/plan-for-production.dita index 2bb123b7ed..9e94afe09c 100644 --- a/content/install/plan-for-production.dita +++ b/content/install/plan-for-production.dita @@ -1,26 +1,51 @@ - Software, Hardware, and Network Requirements - This section provides information to set up Couchbase Server on production environment. - Planning for production involves ensuring that the hardware and software requirements are met, and - the deployment and security consideration are taken into account before installing the server. You - can choose to install Couchbase Server on-premises on supported platforms or on virtualization - platforms. + System Requirements + Couchbase Server has a basic set of installation and networking requirements that apply + to nearly every deployment. -

    Before you begin the actual installation, ensure that you go through the section .

    -

    For installation instructions on different platforms, see:

      -
    • On-premises
      • -
    • Docker container
      • -
    • Virtualization platforms
      • -

    -
    Select Couchbase Server Edition - -

    Couchbase Server comes in two different editions: Enterprise Edition and Community Edition. - Each edition offers different features and levels of support. For a comparison between the - features between Enterprise and Community editions, see . We recommend using the Enterprise Edition for - production environments.

    -
    +

    Deploying Couchbase Server into production requires that the hardware and software + requirements are met, and that the deployment guidelines and security considerations are taken + into account before installation.

    + + Couchbase System Requirements + + + + + + Requirement + Description + + + + + Platform support + Ensure that you are deploying onto a platform that is supported.

     +     + + Hardware requirements + Ensure that your production environment has adequate hardware and system + resources to meet the needs of your deployment.

     +     + + Networking requirements + Review the Couchbase Server networking requirements to ensure that your + environment can accommodate the various components that you plan to deploy.

     +     + + Deployment-specific requirements + Review to + make sure you are following the best practices for your deployment. + + + +
    diff --git a/content/install/pre-install.dita b/content/install/pre-install.dita index 6e757740d0..faf1d332f5 100644 --- a/content/install/pre-install.dita +++ b/content/install/pre-install.dita @@ -1,45 +1,52 @@ - Hardware Requirements - Follow the recommendations for minimum and recommended hardware and software - specifications. + System Resource Requirements + Although resource requirements will largely depend on the size and resource demands of + your Couchbase deployment, there are some minimum and recommended specifications that you should + follow. - - -
    Minimum Specification -

    Hardware and software requirements may vary depending on the machine and operating system.

    - -

    The minimum hardware specifications to install and effectively operate Couchbase Server - are:

    -
      - -
    • Dual-core x86 CPU running at 2GHz.
      • -
    • 4GB RAM (physical).
      • -
    • A block-based storage device (hard disk, SSD, EBS, iSCSI).
      • -
    -

    Network file systems such - as CIFS and NFS are not supported.

    - You can reduce the CPU and RAM resources below the above-specified - minimum for development and testing purposes. The specification can be as low as 1GB of free - RAM beyond operating system requirements and a single CPU core. However, you must use at - least the required minimum specification for production. - -
    -
    - Recommended Specifications

    The recommended hardware specifications to install and effectively operate Couchbase Server include:

    - -
      -
    • Four core 64-bit x86 CPU running at 3GHz; six cores when using Cross Datacenter - Replication (XDCR) and Views.
      • -
    • 16GB RAM (physical).
      • -
    • A local block-based storage device (hard disk, SSD). Network filesystems such as CIFS - and NFS are not supported.
      • -
    -

    These recommended specifications do not take into account your intended workload. You should - follow the sizing guidelines detailed in Sizing Guidelines when deciding on machine specifications to run - Couchbase Server.

    -
    + + + + + + + + + Minimum Specifications + Recommended Specifications + + + + + CPU + 2 GHz dual core x86_64 + 3 GHz quad core x86_64

    3 GHz six core x86_64 when using Cross Datacenter + Replication (XDCR) and Views

     +     + + RAM + 4 GB (physical) + 16 GB (physical) + + + Storage + Block-based (HDD, SSD, EBS, iSCSI)Network file systems such as CIFS and NFS + are not supported. + Block-based (HDD, SSD, EBS, iSCSI)Network file systems such as CIFS and NFS + are not supported. + + + +
    + You can reduce the CPU and RAM resources below the minimum requirements for + development and testing purposes. Resources can be as low as 1GB of free RAM beyond operating + system requirements, and a single CPU core. However, you must use at least the minimum system + requirements for production. +

    The recommended specifications do not take into account your intended workload. You should + follow the sizing guidelines detailed in Sizing + Guidelines when deciding on machine specifications to run Couchbase Server.

    diff --git a/content/install/running-couchbase-in-containers.dita b/content/install/running-couchbase-in-containers.dita new file mode 100644 index 0000000000..2932758712 --- /dev/null +++ b/content/install/running-couchbase-in-containers.dita @@ -0,0 +1,48 @@ + + + + Deploying Couchbase Server in Containers + Couchbase Server is designed to run in the most popular container environments. Use + this topic to explore the different ways that you can deploy Couchbase in + containers. + +

    + + Container Deployment Methods + + + + + + Couchbase Autonomous Operator (Kubernetes) + Docker + + + + + +

    With the Couchbase Autonomous Operator, you can easily automate the deployment of + Couchbase Server on container orchestration platforms like open source Kubernetes + and Red Hat OpenShift.

    +     + +

    Run Couchbase Server using a traditional Docker deployment.

    +     +     + + About the Couchbase Autonomous + Operator + +
      +
      • +
    +     +     + + +
    +

    + +     diff --git a/content/install/sizing-general.dita b/content/install/sizing-general.dita index e62d653461..4d64a8679a 100644 --- a/content/install/sizing-general.dita +++ b/content/install/sizing-general.dita @@ -1,35 +1,38 @@ - Sizing Guidelines Evaluate the overall - performance and capacity requirements and determine the hardware and other resources. + Sizing Guidelines Evaluate the overall performance and capacity goals that you have for Couchbase, and use + that information to determine the necessary resources that you'll need in your + deployment.

    When you plan to deploy a Couchbase Server cluster, perhaps the most common (and important) - question that comes up is: how many nodes do I need and what size do they need to be? To learn - more, contact Couchbase Support for sizing help. You can also read about sizing in How many nodes do I need and what size do they need to + be?

    +

    With the increasing number of Couchbase services and the flexibility of the Couchbase Data + Platform, the answer to this question can be challenging. This guide aims to help you better + size your deployment.

    +

    If you want detailed recommendations for your specific deployment, you can contact Couchbase + Support. You can also find more sizing information on the How many nodes.

    -

    With the introduction of Multi-Dimensional Scaling (MDS), sizing is becoming more challenging, and this - guideline aims to help users better size their clusters.

    + format="html" scope="external">Couchbase Blog.

    + The sizing recommendations and calculations discussed in this guide are based on an + analysis of performance data and common use-cases.
    General Considerations

    The sizing of your Couchbase Server cluster is critical to its overall stability and - performance. While there are obviously many variables that go into this, the idea is to - evaluate the overall performance and capacity requirements for your workload and dataset. - Then divide that into the hardware and resources you have available. Your application wants - the majority of reads coming out of the cache, and the I/O capacity to handle its writes. - There needs to be enough capacity in all areas to support everything the system is doing - while maintaining the required level of performance.

    -

    This guideline will refer to five determining factors one should be aware of when sizing a - Couchbase Server cluster: RAM, Disk (I/O and space), CPU and network bandwidth.

    + performance. While there are obviously many variables, the main point is this: You need to + evaluate the overall performance and capacity requirements for your workload and dataset, + and then divide that into the hardware and resources you have available. Your application + wants the majority of reads to come out of the cache, and to have the I/O capacity to handle + the writes. There needs to be enough capacity in all areas to support everything the system + is doing while maintaining the required level of performance.

    Sizing Couchbase Server Resources -

    There are several resources you need to consider when sizing a Couchbase Server node:

      -
    • Number of CPUs
      • +

      This guide discusses five determining factors that you should be aware of when sizing a + Couchbase Server cluster node:

        +
      • CPU
      • RAM
      • Disk space
      • Network
        • @@ -41,7 +44,8 @@
        CPU
        -
        CPU refers to the number of cores required to run your workload.
        +
        CPU refers to the number of cores and clock speed that are required to run your + workload.
        @@ -49,22 +53,20 @@
        RAM
        RAM is frequently one of the most crucial areas to size correctly. Cached documents allow the reads to be served at very low latency and consistently high throughput.
        -
        This resource represents the main memory allocated to Couchbase Server and should be - determined by the following factors: -
          -
        • How much free RAM is available beyond OS and other applications.
          • -
        • How much data do you want to store in main memory.
          • -
        • How much latency you expect from KV/indexing/query performance.
          • -

        The new components that require RAM are:

          +
          This resource represents the main memory that you allocate to Couchbase Server and + should be determined by the following factors:
            +
          • How much free RAM is available beyond OS and other applications
            • +
          • How much data do you want to store in main memory
            • +
          • How much latency you expect from KV/indexing/query performance
            • +

          Some components that require RAM are:

          • Memory-optimized Global Indexes, which enable in-memory index processing and index - scans with the lowest latency.
            • -
          • Full Text Search (FTS), where the minimum RAM allocation is 512MB, and recommended - is 2048MB+ RAM.
            • + scans with the lowest latency. +
          • Full Text Search (FTS)

          - Minimum RAM quota for Couchbase Server components + Minimum RAM Quota for Couchbase Server Components @@ -77,19 +79,19 @@ Data Service - 1GB + 1 GB Indexing Service powered by ForestDB - 256MB + 256 MB Indexing Service powered by Memory-Optimized Index - 256MB minimum, 1GB recommended. + 256 MB minimum, 1 GB recommended. Full text search - 512MB + 512 MB; 2048+ MB recommended @@ -101,23 +103,24 @@
          Disk space
          Requirements for your disk subsystem are:
            -
          • Disk size, which refers to the amount of the disk storage space - needed to hold your entire data set.
            • -
          • Disk I/O, which is a combination of your sustained read/write rate, - the need for compacting the database files, and anything else that requires disk - access.
            • -

          To better support your Couchbase Server, keep in mind the following:

            -
          • Disk space continues to grow if fragmentation ratio keeps climbing. To mitigate - that, add enough buffer in your disk space to store all the data. Also, keep an - eye on the fragmentation ratio in the UI and trigger compaction process when - needed.
            • -
          • SSD is desired but not required. SSD will bring much better performance than HDD - regarding disk throughput and latency.
            • -

          +
        • Disk size — which refers to the amount of the disk storage space that + is needed to hold your entire data set.
          • +
        • Disk I/O — which is a combination of your sustained read/write rate, + the need for compacting the database files, and anything else that requires disk + access.
          • +

          To better support Couchbase Server, keep in mind the following:

            +
          • Disk space continues to grow if fragmentation ratio keeps climbing. To mitigate + this, add enough buffer in your disk space to store all of the data. Also, keep an + eye on the fragmentation ratio in the Couchbase user interfaces and trigger + compaction processes when needed.
            • +
          • Solid State Drives (SSDs) are desired, but not required. An SSD will give much + better performance than a Hard Disk Drive (HDD) when it comes to disk throughput + and latency.
            • +

          - Minimal and Recommended Hardware Resources + Minimum and Recommended Hardware Resources @@ -132,19 +135,20 @@ CPU - 4 - 8 and above. Both the number and performance of CPUs are important. + 4 cores + 8 cores and above

          Both the number and performance of CPUs are + important.

                     RAM - 4GB - 16GB and above. More RAM means that more data can be served from memory and - allow for better performance. + 4 GB + 16 GB and above

          More RAM means that more data can be served from memory, + leading to better performance.

                     Disk - 8GB - 16GB and above + 8 GB + 16 GB and above @@ -155,8 +159,8 @@
          Network
          Enough network bandwidth is vital to the performance of Couchbase Server. A reliable high-speed network for intra-cluster and inter-cluster communications has a huge effect - on overall performance and scalability of Couchbase Server. Most people can do this with - a 1GB interface, some need 10GB.
          + on overall performance and scalability of Couchbase Server. Most deployments can achieve + optimal performance with 1 Gbps interconnects, but some may need 10 Gbps. @@ -166,184 +170,163 @@ -
          - Data Service Nodes -

          Data Service nodes handle data service operation, such as create/read/update/delete - (CRUD) operations.

          - -

          It is important to keep use cases and application workloads in mind since different - application workloads have different resource requirements. For example, if your working set - needs to be fully in memory, you might need large RAM size. On the other hand, if your - application requires only 5% of data in memory, you will need enough disk space to store all - the data and fast enough disks for your read/write operations.

          -

          You can start sizing the Data Service nodes by asking the following questions:

          -
            -
          1. Is your application primarily (or even exclusively) using individual document access?
            2. -
          2. Do you plan to use Views?
            3. -
          3. Do you plan to use XDCR? Please refer to Couchbase Server 4.0 documentation about different deployment topology.
            4. -
          4. What’s your working set size and what are your data operation throughput or latency - requirements?
            5. -
          - -

          Answers to the above questions can help you better understand the capacity requirement of - your cluster and provide a better estimation for sizing.

          -

          Inside Couchbase Server, we looked at performance data and customer use cases to provide - sizing calculations for each of the areas: CPU, Memory, Disk, and Network.

          -

          A use case for RAM sizing was used as an example with the following data:

          -
          Input variables for sizing RAM - - - - - - Input Variable - value - - - - - documents_num - 1,000,000 - - - ID_size - 100 - - - value_size - 10,000 - - - number_of_replicas - 1 - - - working_set_percentage - 20% - - - -
          - - - Constants for sizing RAM - - - - - - Constants - value - - - - - Type of Storage - SSD - - - overhead_percentage - 25% - - - metadata_per_document - 56 for 2.1 and higher, 64 for 2.0.x - - - high_water_mark - 85% - - - -
          -

          Based on the provided data, a rough sizing guideline formula would be:

          - - Guideline Formula for Sizing a Cluster - - - - - - Variable - Calculation - - - - - no_of_copies - 1 + number_of_replicas - - - total_metadata - (documents_num) * (metadata_per_document + ID_size) * +
          Sizing Data Service Nodes

          Data Service nodes handle data service + operations, such as create/read/update/delete (CRUD).

          It's important to keep use-cases + and application workloads in mind since different application workloads have different + resource requirements. For example, if your working set needs to be fully in memory, you might + need large RAM size. On the other hand, if your application requires only 5% of data in + memory, you will need disks with enough space to store all of the data, and that are fast + enough for your read/write operations.

          You can start sizing the Data Service nodes by + answering the following questions:

            +
          1. Is the application primarily (or even exclusively) using individual document access?
            2. +
          2. Do you plan to use Views?
            3. +
          3. Do you plan to use XDCR?
            4. +
          4. What’s your working set size and what are your data operation throughput and latency + requirements?
            5. +

          Answers to the above questions can help you better understand the capacity requirement + of your cluster and provide a better estimation for sizing.

          The following is an + example use-case for sizing RAM:
          + Input Variables for Sizing RAM + + + + + + Input Variable + Value + + + + + documents_num + 1,000,000 + + + ID_size + 100 + + + value_size + 10,000 + + + number_of_replicas + 1 + + + working_set_percentage + 20% + + + +
          + Constants for Sizing RAM + + + + + + Constants + Value + + + + + Type of Storage + SSD + + + overhead_percentage + 25% + + + metadata_per_document + 56 for 2.1 and higher, 64 for 2.0.x + + + high_water_mark + 85% + + + +

          Based on the provided data, a rough sizing guideline formula would be:

          + Guideline Formula for Sizing a Cluster + + + + + + Variable + Calculation + + + + + no_of_copies + 1 + number_of_replicas + + + total_metadata + (documents_num) * (metadata_per_document + ID_size) * (no_of_copies) - - - total_dataset - (documents_num) * (value_size) * (no_of_copies) - - - working_set - total_dataset * (working_set_percentage) - - - Cluster RAM quota required - (total_metadata + working_set) * (1 + headroom) / + + + total_dataset + (documents_num) * (value_size) * (no_of_copies) + + + working_set + total_dataset * (working_set_percentage) + + + Cluster RAM quota required + (total_metadata + working_set) * (1 + headroom) / (high_water_mark) - - - number of nodes - Cluster RAM quota required / per_node_ram_quota - - - - -
          -

          Based on the above formula, these are the suggested sizing guidelines:

          - - - Suggested Sizing Guideline - - - - - - Variable - Calculation - - - - - no_of_copies - = 1 for original and 1 for replica - - - total_metadata - = 1,000,000 * (100 + 56) * (2) = 312,000,000 - - - total_dataset - = 1,000,000 * (10,000) * (2) = 20,000,000,000 - - - working_set - = 20,000,000,000 * (0.2) = 4,000,000,000 - - - Cluster RAM quota required - = (312,000,000 + 4,000,000,000) * (1+0.25)/(0.85) = 6,341,176,470 - - - -
          - -

          This tells you that the RAM requirement for whole cluster is 7GB. - Note that this amount is in addition to the RAM requirements for the operating system and any - other software that runs on the cluster nodes.

          - - -
    - + + + number of nodes + Cluster RAM quota required / per_node_ram_quota + + + + +

    Based on the above formula, these are the suggested sizing guidelines:

    + Suggested Sizing Guideline + + + + + + Variable + Calculation + + + + + no_of_copies + = 1 for original and 1 for replica + + + total_metadata + = 1,000,000 * (100 + 56) * (2) = 312,000,000 + + + total_dataset + = 1,000,000 * (10,000) * (2) = 20,000,000,000 + + + working_set + = 20,000,000,000 * (0.2) = 4,000,000,000 + + + Cluster RAM quota required + = (312,000,000 + 4,000,000,000) * (1+0.25)/(0.85) = 6,341,176,470 + + + +

    This tells you that the RAM requirement for the whole cluster is 7 GB. Note that + this amount is in addition to the RAM requirements for the operating system and any other + software that runs on the cluster nodes.

    @@ -352,100 +335,79 @@ -
    - Index Service Nodes -

    A node running the Index Service must be sized properly to create and maintain - secondary indexes and to perform index scan for N1QL queries.

    - -

    Similarly to the nodes running the data service, there is a set of questions you need to - answer to take care of your application needs:

    -
      -
    1. What is the length of the document key?
      2. -
    2. Which fields need to be indexed?
      3. -
    3. Will you be using simple or compound indexes?
      4. -
    4. What is the minimum, maximum, or average value size of the index field?
      5. - -
    5. How many indexes do you need?
      6. -
    6. How many documents need to be indexed?
      7. -
    7. How often do you want compaction to run?
      8. -
    -

    Answers to the above questions can help you better understand the capacity requirement of - your cluster and provide a better estimation for sizing. At Couchbase, we looked at - performance data and customer use cases to provide sizing calculations for each of the areas: - CPU, Memory, Disk and Disk I/O.

    -

    Here is a disk size example:

    - - - Disk Sizes - - - - - - Input variable - Value - - - - - docID - 20 bytes - - - Number of index fields - 1 - - - Secondary index - 24 bytes - - - Number of documents to be indexed - 20M - - - -
    -

    When we calculate disk usage for the above test cases, there are a few factors we need to keep in mind:

    -
      -
    1. Compaction is disabled. This case illustrates the worst-case scenario for disk usage.
      2. -
    2. Couchbase Server uses an append-only storage format. Therefore, actual disk usage will be - larger than data size.
      3. -
    3. Fragmentation will affect the disk usage. The larger the fragmentation, the more disk you will need.
      4. -
    -

    Based on our experiment, we noticed that the above index consumes 6GB of disk space.

    - - -
    -
    - Query Service Nodes -

    A node running the Query Service executes queries for your application - needs.

    -

    Since the Query Service doesn’t need to persist data to disk, there are very minimal resource - requirements for disk space and disk I/O. You only need to consider CPU and memory.

    -

    There are a few questions that will help size the cluster:

    -
      -
    1. What types of queries you need to run?
      2. -
    2. Do you need to run stale=ok or stale=false queries?
      3. -
    3. Are the queries simple or complex (requiring JOINs, for example)?
      4. -
    4. What are the throughput and latency requirements for your queries?
      5. -
    -

    Different queries have different resource requirements. A simple query might return results - within milliseconds while a complex query may require several seconds.

    -

    At Couchbase, we looked at performance data and customer use cases to provide sizing - calculations.

    -

    Let’s use CPU/Memory utilization for an example for a Query Service. Assume that you have a - user profile store, which stores user’s name, email and address. You would like to query based - on user’s email address, and you create a secondary index on email. Now you would like to run - a query that looks like this:

    - Select * from bucket where email = "foo@gmail.com" -

    By default, N1QL uses stale=ok for a consistency model.

    -

    To achieve max throughput of this query, we noticed that it utilized 24 cores completely to - achieve an 80% latency of 5ms against a bucket of 20M documents.

    - - -
    +
    Sizing Index Service Nodes

    A node running the Index Service must be + sized properly to create and maintain secondary indexes and to perform index scan for N1QL + queries.

    Similarly to the nodes that run the Data Service, there is a set of questions + you need to answer to take care of your application needs:

      +
    1. What is the length of the document key?
      2. +
    2. Which fields need to be indexed?
      3. +
    3. Will you be using simple or compound indexes?
      4. +
    4. What is the minimum, maximum, or average value size of the index field?
      5. +
    5. How many indexes do you need?
      6. +
    6. How many documents need to be indexed?
      7. +
    7. How often do you want compaction to run?
      8. +

    Answers to these questions can help you better understand the capacity requirement of + your cluster, and provide a better estimation for sizing.

    The following is an example + use-case for sizing disk: + Disk Sizes + + + + + + Input variable + Value + + + + + docID + 20 bytes + + + Number of index fields + 1 + + + Secondary index + 24 bytes + + + Number of documents to be indexed + 20M + + + +

    When you calculate disk usage for the above test cases, there are a few factors you + need to keep in mind:

      +
    1. Compaction is disabled. This case illustrates the worst-case scenario for disk + usage.
      2. +
    2. Couchbase Server uses an append-only storage format. Therefore, actual disk usage will + be larger than data size.
      3. +
    3. Fragmentation will affect the disk usage. The larger the fragmentation, the more disk + you will need.
      4. +

    The above index consumes 6 GB of disk space.

    +
    Sizing Query Service Nodes

    A node that runs the Query Service executes + queries for your application needs.

    Since the Query Service doesn’t need to persist + data to disk, there are very minimal resource requirements for disk space and disk I/O. You + only need to consider CPU and memory.

    There are a few questions that will help size + the cluster:

      +
    1. What types of queries do you need to run?
      2. +
    2. Do you need to run stale=ok or stale=false + queries?
      3. +
    3. Are the queries simple or complex (requiring JOINs, for example)?
      4. +
    4. What are the throughput and latency requirements for your queries?
      5. +

    Different queries have different resource requirements. A simple query might return + results within milliseconds while a complex query may require several seconds.

    The + following is an example use-case for sizing CPU:

    Assume that you have a user profile + store, which stores a user’s name and email address. You would like to query based on a + user’s email address, and you create a secondary index on email. Now you would like to run a + query that looks like this: +

    Select * from bucket where email = "foo@gmail.com"

    By default, + N1QL uses stale=ok for a consistency model.

    It was observed that + this query utilized 24 cores completely to achieve an 80% latency of 5ms against a bucket of + 20M documents.

    diff --git a/content/install/startup-shutdown.dita b/content/install/startup-shutdown.dita index 61e9665344..8fdaf3b6e7 100644 --- a/content/install/startup-shutdown.dita +++ b/content/install/startup-shutdown.dita @@ -7,29 +7,36 @@
    Start and Stop on Linux

    On Linux, Couchbase Server is installed as a standalone application with support for - running as a background (daemon) process during start-up either through the use of - /etc/init.d/couchbase-server, or via systemctl, - depending on the Linux version.

    -

    The startup script is automatically installed during installation from one of the Linux - packaged releases (Debian/Ubuntu or Red Hat/CentOS). By default, Couchbase Server is - configured to be started automatically at run levels 2, 3, 4, and 5, and explicitly shut down - at run levels 0, 1 and 6.

    + running as a background (daemon) process during startup. The startup script is automatically + installed when you use any of the packaged releases of Couchbase Server for Linux.

    +

    For newer versions of Linux that use systemd for init processing, such + as RHEL/CentOS 7, Ubuntu 15.04, and SUSE 12, the startup script uses + systemctl. For older versions of Linux that use + SysV for init processing, the startup script uses + /etc/init.d/couchbase-server. By default, Couchbase Server is + configured to start automatically at run levels 2, 3, 4, and 5, and explicitly shut down at + run levels 0, 1, and 6.

    To manually start Couchbase Server using the startup/shutdown script:

      -
    • For RHEL 6: # sudo service couchbase-server start
      • -
    • For RHEL 7:# sudo systemctl start couchbase-server
      • -
    To manually stop Couchbase Server using the startup/shutdown script:
      For newer Linux (such as RHEL/CentOS 7 and + later):sudo systemctl start couchbase-server +
      +
    • For older Linux (such as RHEL + 6):sudo service couchbase-server start
      • +
    To manually stop Couchbase Server using the startup/shutdown script:
      -
    • For RHEL 6, as a root - user:# sudo service couchbase-server stop
      • -
    • For RHEL 7: # sudo systemctl stop couchbase-server
      • +
    • For newer Linux (such as RHEL/CentOS 7 and later): + sudo systemctl stop couchbase-server
      • +
      +
    • For older Linux (such as RHEL + 6):sudo service couchbase-server stop
    On CentOS, you may see a failure when trying to run service couchbase-server start as a root user: "Failed to start couchbase-server.service: Access denied". This failure could be caused by a bug in systemd. We recommend that you try running - systemctl daemon-reexec and then run service + scope="external">a bug in systemd. We recommend that you try + running systemctl daemon-reexec and then run service couchbase-server start again.

    On Windows diff --git a/content/install/synchronize-clocks-using-ntp.dita b/content/install/synchronize-clocks-using-ntp.dita index 83030fd433..17f102abff 100644 --- a/content/install/synchronize-clocks-using-ntp.dita +++ b/content/install/synchronize-clocks-using-ntp.dita @@ -7,7 +7,7 @@ of NTP is to keep an individual node's clock 'accurate'. This is done by having the node periodically synchronize its clock with a reference server. You can specify multiple servers to provide redundancy in case one or more time servers are unavailable. Correct time - synchronisation is important in Couchbase Server for Expiry behaviour and to maintain accurate + synchronisation is important in Couchbase Server for Expiry behavior and to maintain accurate Hybrid Logical Clocks used in
  • In case of a bigger, scalable cluster, Couchbase recommends that you run your own NTP server pool that is externally synchronized. You should have at least two NTP - sources on the same local network as each Couchbase RZA server group. This - configuration will provide perfect relative and absolute clock synchronization, high - availability, and easy maintenance.
    • + sources on the same local network as each Couchbase server group. + This configuration will provide perfect relative and absolute clock synchronization, + high availability, and easy maintenance.
  • Couchbase recommends configuring NTP servers/pool as peers across clusters. This enables the NTP servers/pool on one cluster to have a relationship between the NTP servers/pool of the other cluster.
    • @@ -62,7 +63,7 @@ do not need to perform this action manually. However, if you need to make adjustments to the NTP configuration, it is useful to know how to stop and start the service.

    For example, the following command starts the daemon: - $ service ntp start -x + $ service ntp start -x
  • Point each instance of NTP to the same set of reference servers.

    Specify the time server(s) in the NTP configuration file (usually /etc/ntp.conf). You can specify multiple servers, one server per line.

    It is recommended to diff --git a/content/install/testing.dita b/content/install/testing.dita index 293de5f496..ac390f3207 100644 --- a/content/install/testing.dita +++ b/content/install/testing.dita @@ -1,43 +1,76 @@ - - - Verifying Your Couchbase Server InstallationTesting the connection to Couchbase Server can be performed in a number of different + + + Verifying the Couchbase Server Installation + Testing the connection to Couchbase Server can be performed in a number of different ways. - -

    Connecting to the Couchbase web console on a node via a web browser provides basic - confirmation that your node is available. You can also use the couchbase-cli - command to query your Couchbase Server node to confirm that the node is available.

    - - The Couchbase web console uses the same port as smart clients when - communicating with Couchbase Server. If you can connect to the Couchbase web console from a - particular machine then administration and database clients on the same machine can also - connect to the core cluster port and perform operations. The Couchbase web console will warn - if the web browser loses connectivity to the node. -

    To verify that clients can connect to your node, you can use either the - cbworkloadgen command, or telnet.

    -

    The cbworkloadgen command is a python program which performs basic - operations (reads and writes) against the Couchbase Server cluster.

    -

    Using telnet only checks the memcached compatibility ports and the memcached - text-only protocol.

    -
    Testing with <cmdname>telnet</cmdname> -

    The simplest method to determine whether Couchbase Server is running is to use - Telnet to connect to the server with the memcached text - protocol.

    -

    Telnet must be installed on your server to connect to Couchbase Server - using this method. It is supplied as standard on most platforms, or can be obtained as a - separate package and installed via your operating system's standard package manager.

    - - You should not use the Telnet method for communicating - with your server within your application. Instead, use one of the Couchbase Server SDKs. - -

    Connect to the server (via the legacy memcached protocol using Moxi):

    - > telnet localhost1 11211 + +
    + Basic Verification +

    Connecting to the Couchbase Server Web Console with a Web browser provides basic + confirmation that a node is available.

    +

    On all platforms, you can access the Web Console by connecting to the embedded Web server + on port 8091. For example, if your server can be identified on your network as + serverA, you can access the Web Console by going to + http://serverA:8091/ or + http://<serverA's-IP-address>:8091/. (If you set up Couchbase + Server on a port other than 8091, go to that specified port.) You should + clear your browser's cache before attempting to access to the Web Console.

    +

    If you are logged into the node itself, you can go to + http://localhost:8091. You can also use the + couchbase-cli command to query node to confirm that it is available.

    + The Web Console uses the same port as smart clients when communicating with + Couchbase Server. If you can connect to the Web Console from a particular machine, then + administration and database clients on the same machine can also connect to the core cluster + port and perform operations. The Web Console will provide a warning if the Web browser loses + connectivity to the node. +
    +
    + Detailed Verification +

    To verify that clients can connect to the node, you can use either cbworkloadgen or telnet.

    +

    The cbworkloadgen command is a python program that performs basic + operations (reads and writes) against the Couchbase Server cluster.

    +

    Using telnet only checks the memcached compatibility ports and the + memcached text-only protocol.

    +
    + +
    Testing With cbworkloadgen
    +
    +

    The command cbworkloadgen generates random data, then performs + reads and writes of this data in Couchbase Server. It is not intended to be a + performance benchmarking tool.

    +

    To test a Couchbase Server installation using cbworkloadgen, + execute the following command, supplying the IP address of the running node:

    + > cbworkloadgen -n [host]:8091 + Thread 0 - average set time : 0.0257480939229 seconds , min : 0.00325512886047 seconds , max : 0.0705931186676 seconds , operation timeouts 0 +

    The progress and activity of the tool can also be monitored within the Web + Console.

    +

    Further details about cbworkloadgen can be found on its CLI reference page.

    +
    +     + +
    Testing With telnet
    +
    +

    The simplest method to determine whether Couchbase Server is running is to use + telnet to connect to the server with the memcached text + protocol.

    +

    Telnet must be installed on your server to connect to Couchbase + Server using this method. Most platforms come with telnet + pre-installed, but it can be obtained as a separate package if necessary.

    + You should not use the telnet method for + communicating with Couchbase Server within your application. Instead, use one of the + Couchbase Server SDKs. +

    Connect to the server (via the legacy memcached protocol using Moxi):

    + > telnet localhost1 11211 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. -

    Make sure it is responding (stats is a great way to check basic health):

    - stats +

    Make sure it is responding (stats is a great way to check basic + health):

    + stats STAT delete_misses 0 STAT ep_io_num_write 0 STAT rejected_conns 0 @@ -50,37 +83,24 @@ STAT pid 23871 ... END -

    Store a document:

    - set test_key 0 0 1 +

    Store a document:

    + set test_key 0 0 1 a STORED -

    Retrieve the document:

    - get test_key +

    Retrieve the document:

    + get test_key VALUE test_key 0 1 a END -

    Disconnect:

    - quit +

    Disconnect:

    + quit Connection closed by foreign host. > -

    All of the memcached protocol's commands will work through Telnet.

    -
    - -
    Testing with <cmdname>cbworkloadgen</cmdname> - -

    The command cbworkloadgen generates random data then performs reads and - writes on this data in Couchbase Server. It is not intended to be a performance benchmarking - tool.

    - -

    To test a Couchbase Server installation using cbworkloadgen, execute the - following command, supplying the IP address of the running node:

    - > cbworkloadgen -n [host]:8091 - Thread 0 - average set time : 0.0257480939229 seconds , min : 0.00325512886047 seconds , max : 0.0705931186676 seconds , operation timeouts 0 -

    The progress and activity of the tool can also be monitored within the Couchbase web - console.

    -

    Further details about cbworkloadgen can be found on its CLI reference page.

    +

    All of the memcached protocol's commands will work through + telnet.

    + + +
    - - - +     + diff --git a/content/install/thp-disable.dita b/content/install/thp-disable.dita index 45cbd0011d..11d692ee9e 100644 --- a/content/install/thp-disable.dita +++ b/content/install/thp-disable.dita @@ -67,18 +67,23 @@ esac
    Register the code in the OS
    As root do the following:
      -
    1. Create a file with the above code: $ sudo vi /etc/init.d/disable-thp
      2. -
    2. Chmod the file to be executable: $ sudo chmod 755 /etc/init.d/disable-thp
      3. -
    3. Execute it so it takes effect right now: $ sudo service disable-thp start -
      4. +
    4. Create a file with the above code: + # vi /etc/init.d/disable-thp
      5. +
    5. Chmod the file to be executable: + # chmod 755 /etc/init.d/disable-thp
      6. +
    6. Execute it so that it takes effect right now: + # service disable-thp startIf you encounter an error + running the script, you can run the commands manually as follows: + # echo never > /sys/kernel/mm/transparent_hugepage/enabled +# echo never > /sys/kernel/mm/transparent_hugepage/defrag{{}}
    Make sure the Init script starts at boot
    -
    For Red Hat variants: $ sudo chkconfig disable-thp on
    -
    For Ubuntu: $ sudo update-rc.d disable-thp defaults
    +
    For Red Hat variants: sudo chkconfig disable-thp on
    +
    For Ubuntu: sudo update-rc.d disable-thp defaults
    @@ -88,7 +93,7 @@ esac
    Check the status of THP by issuing the following commands: cat /sys/kernel/mm/transparent_hugepage/enabled cat /sys/kernel/mm/transparent_hugepage/defrag
    -
    On some Red Hat or Red Hat variants, you might have to do this instead: +
    On some Red Hat variants, you might have to do this instead: cat /sys/kernel/mm/redhat_transparent_hugepage/enabled cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
    For both files, the output should be like this: always madvise [never] diff --git a/content/install/upgrade-matrix.dita b/content/install/upgrade-matrix.dita index 11c94a08d1..fdae1f1a04 100644 --- a/content/install/upgrade-matrix.dita +++ b/content/install/upgrade-matrix.dita @@ -10,7 +10,7 @@ Use the same Couchbase Server version number when upgrading from the Community Edition to the Enterprise Edition. Version differences can result in a failed upgrade. -

    The following table shows the supported upgrade paths to version 5.0: The following table shows the supported upgrade paths to version 5.5:
    @@ -36,25 +36,31 @@ 4.0.0 Enterprise Edition Offline or Online - Direct upgrade to 5.0 + Direct upgrade to 5.5 4.1.x Enterprise Edition Offline or Online - Direct upgrade to 5.0 + Direct upgrade to 5.5 4.5.x Enterprise Edition Offline or Online - Direct upgrade to 5.0 + Direct upgrade to 5.5 4.6.x Enterprise Edition Offline or Online - Direct upgrade to 5.0 + Direct upgrade to 5.5 + + + 5.0.x and 5.1.0 + Enterprise Edition + Offline or Online + Direct upgrade to 5.5 diff --git a/content/install/upgrade-offline.dita b/content/install/upgrade-offline.dita index 32545fd22b..b8701fe5c0 100644 --- a/content/install/upgrade-offline.dita +++ b/content/install/upgrade-offline.dita @@ -33,8 +33,7 @@
  • Click Data Buckets |your_bucket. In the Summary section, check that Disk write queue is equal to 0. If you have more than one data bucket in your cluster, repeat this check on each - bucket.

    • + bucket.
  • Create a backup of your cluster data using the cbbackupmgr tool.
    • diff --git a/content/install/upgrade-online.dita b/content/install/upgrade-online.dita index 9a82d5a796..f8f09477de 100644 --- a/content/install/upgrade-online.dita +++ b/content/install/upgrade-online.dita @@ -65,15 +65,12 @@
  • Select Servers Active Servers - to view and manage the cluster nodes:

    • + to view and manage the cluster nodes.
  • Click Add Server.
  • In the Add Server dialog, provide either a hostname or IP address for the new node to be added. Enter your Couchbase Server administrative credentials in the fields Username and Password and select - the appropriate service.

    + the appropriate service.
  • Remove one of your existing old nodes from the cluster.

    Under Server Nodes diff --git a/content/install/upgrade-strategies.dita b/content/install/upgrade-strategies.dita index 92ac72acc2..1c944c6a2f 100644 --- a/content/install/upgrade-strategies.dita +++ b/content/install/upgrade-strategies.dita @@ -8,56 +8,57 @@ Which way is best for your cluster will depend on a few factors in your architecture and company organization (e.g. SLAs). On this and subsequent pages of this section we will discuss not just the mechanics of an upgrade, but why each may or may work best for you.

    -
    Option #1 - Rolling Online Upgrade +
    Option #1 - Rolling Online Upgrade

    A rolling online upgrade is the recommended upgrade process for a Couchbase Server cluster and should be chosen unless you - have special circumstances that would rule it out. As an online upgrade, this method avoids downtime for the database and - applications because all data operations continue normally throughout the process. Rolling upgrades involve using the - rebalance feature of Couchbase to redistribute the data amongst the nodes in the cluster running the Data Service, - thus keeping the data accessible for the duration of the upgrade. As long as you keep some nodes from the Index and Query Services

    -

    As Couchbase Server is specifically designed to provide fully online upgrades - If you find this is not the case, please report a bug.

    + have special circumstances that would rule it out. This method avoids downtime for the database and + applications because all operations continue normally and high availability can be maintained. Rolling upgrades use Couchbase's + rebalance capability to redistribute the data amongst the nodes of a cluster, keeping it available for the duration of the upgrade. + Prior to version 5.0, special care should be taken when upgrading nodes running the index service.

    +

    Couchbase Server is specifically designed to provide fully online upgrades. If you find this is not the case, please open a support ticket or + report a bug.

    There are three options for rolling online upgrades:

    Swap Rebalance
    This method entails introducing new nodes into a Couchbase Server cluster as you remove an equal number of nodes to be upgraded. - It uses a feature called Swap Rebalance to move vBuckets efficiently - from an existing node running the Data Service to a new node that will run the Data Service, without involving other nodes in the cluster. + It uses a feature called Swap Rebalance to move data efficiently + from the existing nodes to the new nodes, without involving other nodes in the cluster.
    -
    As long as you swap an equal number of nodes (e.g. one for one, two for two, etc.), a Swap Rebalance will be used. +
    As long as you swap an equal number of nodes (e.g. one for one, two for two, etc.), a Swap Rebalance will be triggered. This also keeps the cluster capacity consistent so as to not interfere with the load running on the cluster. - While this method is the safest and provides the most availability, one downside to this option is the overall duration of a - cluster upgrade may be longer as compared to other options. - If the speed of an upgrade is a primary concern for your cluster, as it might be if you have a tight maintenance window, see Graceful Failover or .
    -
    Regular Rebalance
    -
    This method is suitable when you must complete the upgrade using only the nodes currently in the cluster. - It involves removing at least one running node from the cluster, - upgrading the node, and then adding it back in. This upgrade method reduces the capacity of the cluster during the upgrade process.
    +
    Remove and Rebalance
    +
    This method is suitable when you must complete the upgrade using only the nodes currently in the cluster but want to maintain High Availability during the upgrade process. + Since this will reduce the capacity of the cluster, it is important to ensure that there is enough spare capacity across + all necessary resources (disk, CPU, RAM, etc) during the upgrade process. + This process involves removing one running node from the cluster and rebalancing. That node can then be upgraded and used in the Swap Rebalance + upgrade procedure above. When the last node has been upgraded, it can be rebalanced back into the cluster to return to full capacity.
    Like a swap rebalance upgrade, this style will require multiple rebalances to complete.
    -
    Graceful Failover
    +
    Graceful Failover and Delta Recovery
    This option involves performing a rolling online upgrade using a Graceful - Failover instead of the full addition and removal of nodes in option #1. - It is typically faster and less resource intensive because vBuckets do not need to be moved between nodes. - With this process, there is a cluster rebalance but it is only to synchronize the vBuckets on the upgraded - node when each node returns to the cluster. + Failover followed by Delta Recovery + instead of the full addition and removal of nodes in a Swap Rebalance. + It is typically faster and less resource intensive because data does not need to be completely moved between nodes, rather + the replicas are synchronized and activated during the failover and the data resynchronized when the node returns following the upgrade. Another advantage compared to the other online upgrades is that this method preserves the global secondary indexes and doesn’t need to rebuild them.
    -
    The primary downside to this option is increased risk as replicas are used for faster failover and not recreated - until a rebalance occurs. This option is not available when choosing to upgrade with net-new systems +
    The primary downside to this option is decreased high availability as replicas are used for faster failover and not recreated + until the node is returned. This option is not available when choosing to upgrade with net-new systems (as in the case of many cloud deployments) since those new nodes would not have the previous nodes’ data in place. Use Option #1 when upgrading with net-new systems.
    @@ -65,32 +66,27 @@
    -
    Option #2 - Upgrade Using Inter-cluster Replication -

    For this option, another Couchbase Server cluster is created or already exists and then - inter-cluster replication is enabled via the Cross Datacenter Replication (XDCR) built-in feature of Couchbase - Server. The administrator then transitions the application to use the new cluster as part of - the upgrade process. While this upgrade process is relatively straightforward to set up, it +

    Option #2 - Upgrade Using XDCR +

    For this option, another Couchbase Server cluster is created (or already exists) and connected via + Cross Datacenter Replication (XDCR). + The application is transitioned to use one cluster while the other(s) is/are upgraded. + While this upgrade process is relatively straightforward to set up, it requires more investment in servers/instances and networking, as well as a change to the - application so that it uses the newly upgraded cluster. It is best used when an - administrator needs an immediate rollback position or does not want to tinker with the - original cluster for some reason. This method is recommended when an existing XDCR - connection is already in place.

    -

    Some people modify their application, to make this can be a zero downtime upgrade by having - their application aware of both clusters and can failover between them automatically. If you - do this, keep in mind that unlike regular operations against the Data Service, XDCR is - eventually consistent. Most people treat this as an offline upgrade and will roll the - application over with the new cluster configurations. Which solution is best will depend on - your SLA requirements.

    - + application so that it can switch between clusters. It is best used when an existing XDCR connection is already in + place, the application needs a live rollback option or the original cluster cannot be upgraded online for some reason. +

    +

    Keep in mind that XDCR is eventually consistent between clusters and so care should be taken when + switching the application from one to the other.

    +

    For the fastest upgrade, the unused cluster(s) can be upgraded following the offline upgrade steps below + (if not installed anew)

    Option #3 - Offline Upgrade

    Choose an offline upgrade when the situation calls for an easy and fast upgrade method as well as when the database can incur a controlled outage. The offline upgrade is more likely to succeed in situations where an online upgrade option might fail, but also the rare time a cluster is unstable and has been determined that a Couchbase Server upgrade will fix a specific issue.

    -

    The offline upgrade is the best choice in situations where an online upgrade option might fail, such as the rare times a cluster - is unstable and it has been determined that an upgrade will fix the issue.

    - +

    This procedure involves upgrading one or more nodes without removing them from the cluster. In some cases the whole cluster + may be shut down, upgraded and restarted.

    +

    It is recommended to disable auto-failover before using this method and to re-enable it once complete.

    Choosing the Upgrade Strategy @@ -98,15 +94,17 @@

    Both the online and offline upgrade processes have trade-offs. The following table illustrates some important aspects of the two upgrade strategies.

    • - Differences between online and offline upgrades - + Differences between upgrades + + Feature Online upgrade + XDCR upgrade Offline upgrade @@ -114,6 +112,7 @@ Applications remain available Yes + Yes No @@ -121,16 +120,19 @@ Cluster stays in operationYesNo + No Cluster must be shut down No Yes + Yes - Typically required steps - Rebalance, upgrade, rebalance per node - All cluster nodes are upgraded at once. + Typical steps + Rebalance, upgrade, rebalance + Switch to XDCR cluster, upgrade, switch back + Upgrade one or more nodes without removing from cluster. diff --git a/content/install/upgrade-strategy-for-features.dita b/content/install/upgrade-strategy-for-features.dita new file mode 100644 index 0000000000..0b3991e9da --- /dev/null +++ b/content/install/upgrade-strategy-for-features.dita @@ -0,0 +1,273 @@ + + + + Upgrading to Couchbase Server 5.5 + +

    Upgrading to the latest version of Couchbase Server helps you leverage the latest features + and keeps your clusters running well. The following table summarizes the new features in this + release and provides a handy reference to help you upgrade to this release.

    +

    +

    + Couchbase Server 5.5 Features and Upgrade Strategy + + + + + + + + + + + Feature + Upgrade Level + Upgrade + Type + Upgrade + Path + Mixed Mode + Dependencies + Additional Information + + + + + Data compression - data service + Cluster + Offline/Online + Direct + No + None + None + + + Data compression - XDCR + Cluster + Offline/Online + Direct + No + None + None + + + N1QL Auditing + Cluster + Offline/Online + Direct + No + None + None + + + N1QL TCO/Performance comparison + Node + Offline/Online + Direct + Yes + None + None + + + N1QL backfill configurability + Node + Offline/Online + Direct + Yes + None + None + + + ANSI JOINS + Node + Offline/Online + Direct + Yes + None + None + + + CURL() Whitelisting + Node + Offline/Online + Direct + Yes + None + None + + + ALTER INDEX + Cluster + Offline/Online + Direct + No + All the query and index services nodes must be on version 5.5 + None + + + Index Partitioning + Index service - Cluster

    Query Service - Node

    +     + Offline/Online + Direct + Index service - No

    Query service - Yes

     + All the index service nodes must be on version 5.5, and at least one query node + must be on version 5.5 + None +     + + Aggregate Pushdown + Index service - Cluster

    Query Service - Node

     + Offline/Online + Direct + Index service - No

    Query service - Yes

     + All the index service nodes must be on version 5.5, and at least one query node + must be on version 5.5 + None +     + + Meta() xattrs + Cluster + Offline/Online + Direct + No + Both index and query nodes must be on version 5.5 + Without the upgrade, the META().xattrs will evaluate to MISSING, leading to + wrong results. + + + Analytics Service (Developer Preview) + Cluster + Offline/Online + Direct + No + The cluster should have only one build version. + None + + + Eventing Service + Cluster + Offline/Online + Direct + No + All data and eventing nodes must be running version 5.5. + None + + + Full Text Search - Improved Indexing Implementation (Developer Preview) + Cluster + Offline/Online + Direct + No + None + Newly created indexes will use the new format. + + + Response Time Observability + Cluster + Offline/Online + Direct + No + None + None + + + IPv6 + Cluster + Offline or Online using unidirectional replication + Direct + No + Operating system must support dual stack. + See for details. + + + x.509 client authentication + Cluster + Offline/Online + Direct + No + None + None + + + Log redaction + Cluster + Offline/Online + Direct + No + None + None + + + SCRAM-SHA for non-TLS XDCR connections + Cluster + Offline/Online + Direct + No

    In the mixed mode, non-TLS connections will fall back to using certificate + for authentication.

     + None + None +     + + Max TTL + Cluster + Offline/Online + Direct + No + None + None + + + Enhancements to cbbackupmgr + Cluster + Offline/Online + Direct + No + None + None + + + Doc Editor + Node + Offline/Online + Direct + Yes + None + None + + + +
    +

    +
    Migrating Data from an IPv4 Cluster to an IPv6 + ClusterYou can migrate data from an IPv4 cluster to an IPv6 cluster with an offline + upgrade, or online by using unilateral replication. You must ensure that your operating system + has dual stack support.

    Offline Upgrade from an IPv4 Cluster to an IPv6 + Cluster

    Prerequisites: Ensure that the source cluster is on an operating + system with dual stack support.

    Follow the instructions available in the section . When all the nodes are offline, restart + each node with IPv6 enabled. See for instructions + to enable IPv6.

    Online Upgrade from an IPv4 Cluster to an IPv6 Cluster using + XDCR

    Prerequisites: Ensure that the source cluster is running Couchbase + Server 5.5 on an operating system with dual stack support, and the destination cluster is + running Couchbase Server 5.5 on an IPv6 network.

    Steps:

      +
    1. Upgrade the source cluster to Couchbase Server version 5.5.
      2. +
    2. Create a new cluster with IPv6 enabled and configured. See for instructions to enable IPv6. Note that this + cluster needs to be sized appropriately for the workload, but does not need to be + identical to the source cluster.
      3. +
    3. Create a cluster reference and + replication stream from the source to the destination cluster. At this point, + the destination cluster is not receiving any application load directly, other than the + replication traffic from the source cluster.
      4. +
    4. Monitor the XDCR queue from the source until all mutations are replicated to + the destination cluster.
      5. +
    5. Reconfigure the application to start accessing the destination cluster.
      6. +
    6. Once all your applications have been moved, you can decommission the source + cluster.
      7. +

    IPv4 and IPv6 clusters cannot be paired for bi-directional + (active-active) XDCR replication. For bi-directional replication, both clusters need to be + using IPv4 or IPv6.
    + + diff --git a/content/install/upgrade.dita b/content/install/upgrade.dita index d151074d04..a5fb6009fc 100644 --- a/content/install/upgrade.dita +++ b/content/install/upgrade.dita @@ -3,9 +3,12 @@ Upgrading Couchbase Server Periodically upgrading Couchbase Server helps maintain a properly functioning cluster and offers new features. Upgrading is a relatively simple process that usually incurs zero downtime. -

    Upgrading Couchbase Server regularly is vital to keep your cluster running well. New releases of Couchbase Server provide both bug fixes and new features. For each release, read the release notes to ascertain if there are bugs that might affect a running cluster or features that might be beneficial.

    -

    For an overview of the new features in Couchbase Server, check out the release notes.

    +

    Upgrading Couchbase Server regularly is vital to keep your cluster running well. New releases + of Couchbase Server provide both bug fixes and new features. For each release, read the release notes to ascertain if + there are bugs that might affect a running cluster or features that might be beneficial. + lists the new features + in this release and their upgrade behavior.

    When to do an Upgrade

    If you can, it is best to upgrade during off hours. Avoid upgrading during the system’s peak usage times. Even though rebalances are designed to be a background process, they can impact performance. @@ -15,18 +18,131 @@

    Supported Upgrade Paths - -

    See +

    Couchbase Server supports upgrade paths from one major version to another.

    +

    Be aware of the currently supported versions and versions at or near their end of life. The + support information is available in the Couchbase Support Policy.

    +

    Although infrequent, there may be times when an upgrade to an intermediate version is required before getting to the latest release.

    +

    The following table shows the supported upgrade paths to version 5.5: + + + + + + + + Current Version + Edition + Upgrade Option + Support + + + + + + 4.0.0 + Enterprise Edition + Offline or Online + Direct upgrade to 5.5 + + + 4.1.x + Enterprise Edition + Offline or Online + Direct upgrade to 5.5 + + + 4.5.x + Enterprise Edition + Offline or Online + Direct upgrade to 5.5 + + + 4.6.x + Enterprise Edition + Offline or Online + Direct upgrade to 5.5 + + + 5.0.x and 5.1.0 + Enterprise Edition + Offline or Online + Direct upgrade to 5.5 + + + +

    +

    The following table shows some commonly supported upgrade paths:

    +

    + + Couchbase Server upgrade matrix + + + + + + + Current version + Target version + Process + + + + + 2.5.x + 3.x + Upgrade from version 2.5.x directly to the latest version 3.x using any + supported upgrade strategy. + + + 3.x + 4.x + Upgrade from the latest version 3.x directly to version 4.x using any supported + upgrade strategy. + + + 2.5.x + 4.x + Upgrade from the latest version 2.5.x directly to version 4.x using any + supported upgrade strategy. + + + 4.0 + 4.x (> 4.0) + Services require more planning during an upgrade. + + + 4.x + 5.x + Services require more planning during an upgrade. + + + +
    +

    + +
    Preparing for an Upgrade -

    Before upgrading your production cluster, be sure to test the upgrade process in a test environment that matches your production environment as closely as possible.

    -

    For each upgrade type mentioned above, there is an example checklist to help you start.

    +

    Before upgrading your production cluster, be sure to test the upgrade process in a test + environment that matches your production environment as closely as possible.

    + When upgrading from the Community Edition to the Enterprise Edition, + ensure that you use the same Couchbase Server version number. Version differences can result + in a failed upgrade. +
    @@ -39,13 +155,7 @@ example, each workload is isolated to a set of nodes. There are special considerations for most online upgrade options in an MDS cluster. For more information on this, see . - + -->
    Precautions for Couchbase Mobile’s Sync Gateway

    Take special precaution when upgrading Couchbase Server if you use the cluster in conjunction with Do I need to uninstall and reinstall, or just upgrade the Couchbase Server package?

    For all platforms except Mac OS X, you only need to upgrade the package to the new version. Mac OS X may not upgrade successfully without an uninstall and reinstall. For - uninstallation instructions, see .
    diff --git a/content/integration.ditamap b/content/integration.ditamap index 4389a620f5..4280acb053 100644 --- a/content/integration.ditamap +++ b/content/integration.ditamap @@ -4,11 +4,10 @@ Integrations + - - - + diff --git a/content/introduction.ditamap b/content/introduction.ditamap deleted file mode 100644 index ebbfa1622d..0000000000 --- a/content/introduction.ditamap +++ /dev/null @@ -1,10 +0,0 @@ - - - - Introduction - - - - - - diff --git a/content/introduction/contact-couchbase.dita b/content/introduction/contact-couchbase.dita new file mode 100644 index 0000000000..f6a03fab47 --- /dev/null +++ b/content/introduction/contact-couchbase.dita @@ -0,0 +1,195 @@ + + + + + + Contact Couchbase + + + + Provide feedback, and get help with any problem you may encounter. + + + +
    + + + Options for Making Contact + + +

    + You can get in touch with Couchbase in any of the following ways: +

    + +
      +
    • +

      + Contact Couchbase Support: If you are a customer of Couchbase Server Enterprise Edition, + and have a support contract, you can log into the Couchbase Customer Support Portal, and so make + contact with Couchbase staff-members, who will provide guidance and assist with any problems you + are encountering. Email and phone support are also available. +

      +
      • + +
    • +

      + Participate in a Forum: Multiple online forums are available, whereby you can post a + question, and get answers; both from Couchbase employees and from members of the Couchbase + community of developers and administrators. +

      +
      • + +
    • +

      + Submit a ticket: Couchbase lets you use JIRA software, to report issues as individual + tickets. +

      + +
      • + +
    • +

      + Submit documentation changes: If you find errors in Couchbase documentation, you can + make your own edits to the source; and directly submit your changes to the Couchbase documentation + team. +

      + +
      • +
    + +

    + Further details are provided below. +

    + +
    + +
    + + + Contact Couchbase Support + + +

    + The + Couchbase Support Portal provides online + support for all customers of Couchbase Server Enterprise Edition who have a support contract: +

    + +

    + +

    + +

    + You can also contact Couchbase Support by phoning +1-650-417-7500, or by sending email + to support@couchbase.com. +

    + +
    + +
    + + + Participate in a Forum + + +

    + Couchbase Community Forums let you share knowledge with + developers, administrators, and Couchbase in-house experts. All forums can be accessed at + forums.couchbase.com: +

    + +

    + +

    + +

    + To participate fully, you need to register: left-click on the Sign Up button, at the upper right. + Then, to access individual forums, select from the all categories pull-down menu. +

    + +
    + +
    + + + Submit a Ticket + + +

    + If you have a JIRA account, you can submit tickets to Couchbase. When submitting, include all + information that might be appropriate. For example: +

    + +
      + +
    • +

      + A description of your environment (package installation, build number, operating system, etc). +

      + +
      • + + +
    • +

      All steps required to reproduce the issue. This should include all relevant + expression-names, parameters, flags, and values; with sample output in verbose mode from + repeated executions, using different values.

      +
      • + +
    • +

      + Logs from all cluster-nodes. +

      +
      • + + +
    • +

      + Query-results from the command-line, when failures have occurred through use of the UI; and vice versa. +

      +
      • + +
    +
    + +
    + + + Submit Documentation Changes + + +

    + If you find a documentation-error, to fix it, you can left-click on the link at the upper right of each page: +

    + +

    + +

    + +

    + This takes you to an interactive edit-pane, in which the page-source can be directly edited: +

    + +

    + +

    + +

    + When you have made your edits, left-click on the Commit changes button, + at the lower left: +

    + +

    + +

    + +

    + This allows you to open a pull request, which will be reviewed by Couchbase staff-members, prior + to integration of your submitted changes into the documentation-set. +

    + + +
    + + +     diff --git a/content/introduction/editions.dita b/content/introduction/editions.dita index 7c26e68a3a..c7de8e7777 100644 --- a/content/introduction/editions.dita +++ b/content/introduction/editions.dita @@ -6,7 +6,7 @@ Couchbase Server is available in several editions: enterprise, community, and open source. -

    Each edition offers different features and levels of support. For more information on each feature, see the Enterprise and community edition feature matrix.

    +

    Each edition offers different features and levels of support.

    Enterprise Edition
    @@ -14,9 +14,9 @@ Couchbase Server. We recommend the enterprise edition binaries for commercial production systems running Couchbase Server. The enterprise edition contains some unique features that make it the best fit for production - deployments running in data centers or a public cloud infrastructure. These + deployments running in data centers or a public cloud infrastructure. Use of EE in development and test environments is free. EE requires a paid subscription for production use. Our paid annual subscriptions for EE include a @@ -48,523 +48,9 @@
    -

    The following table compares the features available in the Enterprise and Community - editions:

    - - - Enterprise and community edition feature matrix - - - - - - - Feature - Enterprise Edition - Community Edition - - - - - Software updates and support - - - - Frequent - releases with quality improvements - - - - - Worldwide 24x7 - support - - - - - Patches and maintenance - updates - - - - - Proactive - services - - - - - - High availability & disaster recovery - - - - Intra-cluster - replication - - - - - Auto - failover - - - - - Online - rebalance - - - - - Cross datacenter replication - (XDCR) - - - - - Filtering - replication with XDCR - - - - - Rack and - availability zone awareness - - - - - Basic backup and - restore - - - - - High - performance enterprise backup and restore - - - - - Incremental - and cumulative backup and restore - - - - - - - - - - Security - - - - Built-in account - management - - - - - Role based access - control - - - - - LDAP integrated - account management - - - - - Encrypted - administrative access - - - - - Encrypted - data and query access - - - - - - Encrypted XDCR - - - - - - Encryption with x509 - certificates - - - - - Security - auditing - - - - - - - - - - - - - - - Data access - - - - Core data access - APIs ( Get/Set Operations) - - - - - Tunable consistency - with queries (with staleness setting) - - - - - Tunable durability - (with persistence and replication) - - - - - SQL-like queries with - N1QL - - - - - Query and indexing with - standard GSI - - - - - Query and - indexing with MapReduce views - - - - - Query and - indexing with spatial views - - - - - Native SDKs - - - - - Development and Administration Tools - - - - Visual Web Console for - development - - - - - Visual Web Console for - administration - - - - - Query - editing with smart auto-complete - - - - - Smart schema inference - for document data - - - - - Programmable REST API - access - - - - - Programmable Command line tools - (CLI) - - - - - Performance and scaling - - - - High performance - concurrent query execution - - - - - Query and indexing with - memory-optimized GSI - - - - - Multi-dimensional scaling - - independent scalability - - - - - Homogeneous - scaling - - - - - - -
    -
    Feature Descriptions -
    - -
    Frequent Releases with Quality Improvements
    -
    Couchbase Server Enterprise Edition ships frequent updates to the product with latest - feature enhancements and quality improvements. Community Edition releases - run behind Enterprise Edition by several months.
    -     - -
    Worldwide 24x7 Support
    -
    Customers can only get official 24x7 support worldwide for Couchbase Server Enterprise - Edition.
    -     - -
    Patches and Maintenance Updates
    -
    Couchbase Server quality is continuously improved with maintenance releases and patches. - Enterprise Edition customers can receive frequent maintenance releases and - patches for quality issues they experience. Community edition does not - receive these updates.
    -     - -
    Proactive Services
    -
    Enterprise Edition customers can receive proactive services from Couchbase for sizing and - capacity planning for production deployments, data model and code reviews - and architecture reviews for applications.
    -     - -
    Intra-Cluster Replication
    -
    Intra-cluster replication provides protection against node failures within the cluster. Replication between Couchbase Server nodes is included in both Enterprise and Community Edition.
    -     - -
    Auto Failover
    -
    The smart cluster manager built into Couchbase Server detects and recovers from node failures using auto-failover. Manual and auto-failover between Couchbase Server nodes is included in both Enterprise and Community Edition.
    -     - -
    Online Rebalance
    -
    Online rebalance redistributes the load across nodes as Couchbase Server cluster expands and shrinks by adding and removing nodes. The ability to add or remove nodes and rebalance the cluster data distribution among Couchbase Server nodes is included in both Enterprise and Community Edition.
    -     - -
    Cross Data Center Replication (XDCR)
    -
    Globally distributed applications use XDCR to replicate their Couchbase Server data across multiple data-centers. The ability to replicate data across data centers, between multiple Couchbase Server clusters, is included in both Enterprise and Community Edition.
    -     - -
    Filtering Replication with Cross Data-Center Replication
    -
    Interactive applications use XDCR to improve data locality by deploying data closer to the - users geographies. With XDCR, customers in Asia vs customers in US can have - local copies of the relevant data in their regional data centers. XDCR with - Filtering ensures only the relevant data is carried between clusters across - the wide area network (WAN). This improves replication efficiency and saves - bandwidth. The ability to replicate data selectively with XDCR - Filtering between multiple Couchbase Server clusters is only included in the - Enterprise Edition.
    -     - -
    Rack/Availability Zone Awareness
    -
    Administrators can use Rack and Zone Awareness (server groups) in a Couchbase Server deployment to ensure smart placements of replicas across racks and availability zones. Rack and Zone Awareness with Couchbase Server places replicas smartly to protect against rack or availability-zone failures in public or private cloud deployment. This protection is only available in the Enterprise Edition.
    -     - -
    Basic Backup and Restore
    -
    Backup and Restore ensure full protection against disasters that can take out the entire cluster. Couchbase Server comes built in with online backup and restore tools in both Enterprise and Community Edition.
    -     - -
    High Performance Enterprise Backup and Restore
    -
    Big data applications store many TBs of data and backup and restore operations can be time consuming. With high-performance enterprise backup restore tool, administrators can back up and restore data at a much higher pace and minimize downtime and improve business continuity and disaster recovery. Enterprise backup and restore tool is only included in the Enterprise Edition.
    -     - -
    Incremental and Cumulative Backup and Restore
    -
    With large databases it is important to be able to perform backups and restores incrementally to minimize the restore time and efficiently archiving backups. Incremental and cumulative backups and restores allow creating backup chains and are only available in Enterprise Edition.
    -     - -
    Timestamp-based Conflict Resolution
    -
    Couchbase Server comes with a new option to resolve conflicts with XDCR - using timestamps. With this option, conflicts are resolved by comparing - timestamps of conflicting documents. This option is only available in - Enterprise Edition. -
    -     - -
    Build-in Account Manager
    -
    Couchbase Server comes built in with password protection for administration and data access. Basic account management comes in both Enterprise and Community Editions.
    -     - -
    Role Based Access Control
    -
    Administrators in Couchbase Server can be restricted to specific roles that can manage nodes, buckets, security administration and more. Only the Enterprise Edition comes with role based access control.
    -     - -
    LDAP Integrated Account Management
    -
    Couchbase Server can be integrated with central account management systems within the enterprise through LDAP. Only Enterprise Edition comes with integration for LDAP.
    -     - -
    Encrypted Administrative Access
    -
    Couchbase Server administrators can use encrypted communication for managing public and private cloud deployments for complying with security requirements. Only Enterprise Edition comes with encrypted communication capabilities for cluster administration.
    -     - -
    Encrypted Data and Query Access
    -
    Couchbase Server applications can use encrypted communication for data access and queries for complying with security requirements. Only Enterprise Edition comes with encrypted communication capabilities for query and data access.
    -     - -
    Encrypted Cross Data Center Replication
    -
    Couchbase Server applications using XDCR replication typically use shared network infrastructure across data centers. Cross data-center replication can use encrypted communication for complying with security requirements. Only Enterprise Edition comes with encrypted communication capabilities for XDCR.
    -     - -
    Encryption with x509 Certificates
    -
    Couchbase Server encryption can be managed with built in certificates or with custom certificates from other certificate authorities. Only Enterprise Edition can use custom public or private certificate authorities for encryption of communication
    -     - -
    Security Auditing
    -
    To comply with security requirements and rules, Couchbase Server provides audit trails for all administrative actions. Only Enterprise Edition comes with auditing capabilities.
    -     - -
    Pluggable Authentication Modules
    -
    Pluggable Authentication Modules (PAM) in Couchbase Server enables you to centralize and - synchronize password management across servers. Only Enterprise Edition - comes with PAM capabilities.
    -     -
    Secret Management
    -
    Couchbase Server provides a way to securely manage server secrets which helps hardening of - Couchbase Server. Only Enterprise Edition comes with secret management - capabilities.
    -     - -
    Core Data Access APIs (Get/Set Operations)
    -
    Applications can use core data access APIs to communicate with Couchbase Server. Both Enterprise and Community Edition come built in with full compatibility and support for all data access APIs.
    -     - -
    Tunable Consistency with Queries (with Staleness Setting)
    -
    Couchbase Server provides full consistency for basic data operations but queries can tune consistency from eventual to strict. Both Enterprise and Community Editions come with built in ability to tune consistency at query time.
    -     - -
    Tunable Durability (with Persistence and Replication)
    -
    Couchbase Server provides tunable durability for mutations coming into the system. Couchbase Server core data operations can ask for disk based or replication based durability of data during writes. Both Enterprise and Community Editions comes with full support for tuning durability.
    -     - -
    SQL-like Queries with N1QL
    -
    Couchbase Server N1QL provides full support for SQL-like queries. Both Enterprise and Community Edition come with full support for N1QL for applications.
    -     - -
    Query and Indexing with Standard Global Secondary Indexes
    -
    N1QL queries take advantage of indexing for fast queries. Both Enterprise and Community Editions come built in with standard global secondary indexes.
    -     - -
    Query and Indexing with Memory-Optimized Global Secondary Indexes
    -
    Memory-optimized global secondary indexes provide much faster indexing and query performance for big data applications. Only Enterprise Edition comes with memory-optimized indexes for N1QL.
    -     - -
    Query and Indexing with MapReduce Views
    -
    Map Reduce Views provide local indexes for reporting applications and dashboards. Both Enterprise and Community Edition come with query and indexing with map reduce views.
    -     - -
    Query and Indexing with Spatial Views
    -
    Spatial views provide R-tree indexes for location aware applications. Both Enterprise and Community Editions come with spatial indexes.
    -     - -
    Native SDKs
    -
    Native SDKs make data access simple, fast, highly available and resilient. Both Enterprise and Community Editions come with native SDK support.
    -     - -
    Visual Web Console for Development
    -
    Couchbase Server comes with simple visual and command line tools for development. Both Enterprise and Community Editions comes with built in visual and command line tools.
    -     - -
    Query Editing with Smart Auto-complete
    -
    Auto-complete provide simplified query editing experience while developing queries. Only Enterprise Edition comes built in with query editor with auto-complete.
    -     - -
    Smart Schema Inference for Document Data
    -
    JSON documents in Couchbase Server provide great flexibility without schema management headache that is common in relational databases. With smart Schema Inference and INFER command, developers can easily discover schema including data model, data types and data distribution. Only Enterprise Edition comes with smart schema inference.
    -     - -
    Visual Web Console for Administration
    -
    Couchbase Server comes with simple visual web console for cluster administration. Both Enterprise and Community Editions comes with built in visual web console.
    -     - -
    Programmable REST API Access
    -
    Couchbase Server comes with REST API access for cluster administration. Both Enterprise and Community Editions comes with built in REST APIs for administration.
    -     - -
    Programmable Command Line Tools (CLI)
    -
    Couchbase Server comes with command line access for cluster administration. Both Enterprise and Community Editions comes with built in command line tools for administration.
    -     - -
    High Performance Concurrent Query Execution
    -
    Couchbase Server query service achieves great query throughput and query latency by taking - advantage of large number of queries concurrently and by parallelizing query - execution across multiple cores. Only Enterprise Edition comes with full - parallelism and concurrency support. Community edition can only be deployed - in homogeneous deploy model and comes with limited concurrency and - parallelism (max parallelism can be 4) on each node.
    -     - -
    Multi-dimensional Scaling - Independent Scalability
    -
    Couchbase Server comes with built in services to support independent scaling of various - distinct workloads. Core data operations, indexing, search and query - execution can be deployed across all nodes or can be deployed into - independent zones within the same cluster. Independent deployment of - services achieves both preventing interference between services and - independent scaling of each service. Administrators can pick the right - hardware for each independent zone within the same cluster to add more - memory for indexing and query execution or o provide higher horse-power for - disk IO for fast data access. Only Enterprise Edition comes with the option - to deploy services independently. Both Enterprise and Community Edition - comes with the ability to deploy all services to all nodes - homogeneously.
    -     - -
    Homogeneous Scaling
    -
    Couchbase Server comes with built in services to support independent scaling of various - distinct workloads. Core data operations, indexing, search and query - execution can be deployed across all nodes or can be deployed into - independent zones within the same cluster. Independent deployment of - services achieves both preventing interference between services and - independent scaling of each service. Administrators can pick the right - hardware for each independent zone within the same cluster to add more - memory for indexing and query execution or o provide higher horse-power for - disk IO for fast data access. Only Enterprise Edition comes with the option - to deploy services independently. Both Enterprise and Community Edition - comes with the ability to deploy all services to all nodes - homogeneously.
    -     -
    -
    +

    For a feature comparison between the editions, take a look at https://www.couchbase.com/products/editions.

    diff --git a/content/introduction/images/commitChangesButton.png b/content/introduction/images/commitChangesButton.png new file mode 100644 index 0000000000..505ea03ead Binary files /dev/null and b/content/introduction/images/commitChangesButton.png differ diff --git a/content/introduction/images/couchbase-5.5-beta-sdk-versions.png b/content/introduction/images/couchbase-5.5-beta-sdk-versions.png new file mode 100644 index 0000000000..cac45ae685 Binary files /dev/null and b/content/introduction/images/couchbase-5.5-beta-sdk-versions.png differ diff --git a/content/introduction/images/couchbase-5.5-sdk-versions.png b/content/introduction/images/couchbase-5.5-sdk-versions.png new file mode 100644 index 0000000000..59baa95661 Binary files /dev/null and b/content/introduction/images/couchbase-5.5-sdk-versions.png differ diff --git a/content/introduction/images/editOnGitHub.png b/content/introduction/images/editOnGitHub.png new file mode 100644 index 0000000000..3b9a10acd8 Binary files /dev/null and b/content/introduction/images/editOnGitHub.png differ diff --git a/content/introduction/images/forums-all-categories.png b/content/introduction/images/forums-all-categories.png new file mode 100644 index 0000000000..68705003ef Binary files /dev/null and b/content/introduction/images/forums-all-categories.png differ diff --git a/content/introduction/images/forums.png b/content/introduction/images/forums.png new file mode 100644 index 0000000000..4867926fb4 Binary files /dev/null and b/content/introduction/images/forums.png differ diff --git a/content/introduction/images/gitHubEditPane.png b/content/introduction/images/gitHubEditPane.png new file mode 100644 index 0000000000..224111c326 Binary files /dev/null and b/content/introduction/images/gitHubEditPane.png differ diff --git a/content/introduction/images/supportal.png b/content/introduction/images/supportal.png new file mode 100644 index 0000000000..a4d385466b Binary files /dev/null and b/content/introduction/images/supportal.png differ diff --git a/content/introduction/intro.dita b/content/introduction/intro.dita index 0de0e4fe90..52dcf0c739 100644 --- a/content/introduction/intro.dita +++ b/content/introduction/intro.dita @@ -5,7 +5,7 @@ Why Couchbase?

    Today's mission-critical applications demand support for millions of interactions with - end-users. Traditional databases were build for thousands. Designed for consistency and + end-users. Traditional databases were built for thousands. Designed for consistency and control, they lack agility, flexibility, and scalability. To execute multiple use cases, organizations are forced to deploy multiple types of databases, resulting in a "database sprawl" - and inefficiency, sluggish time to market, and poor customer experience. diff --git a/content/introduction/whats-new.dita b/content/introduction/whats-new.dita index 7d4bdc398a..f137079a89 100644 --- a/content/introduction/whats-new.dita +++ b/content/introduction/whats-new.dita @@ -3,350 +3,227 @@ What's New? -

    To maintain competitive edge, organizations need to constantly revolutionize their customer - experience. To make this possible, Couchbase 5.0 provides the first true Engagement - Database.

    -

    Couchbase Server 5.0 provides unmatched agility and manageability, and delivers unparalleled - performance at any scale. This release provides some exciting features in the areas of - scalability, high-availability, performance, security, indexing and querying, search, tools, - platforms and application development. It also supports a wide ecosystem of big data and - integration connectors.

    -
    - Secure Platform -

    Role Based Access Control (RBAC)

    -

    Building on the Role Based Access Control (RBAC) security model introduced in version 4.5 - for Administrators, Couchbase Server 5.0 introduces RBAC for applications. This allows you - to control, at both broad and granular levels, what administrators, end-users, and - application services can do. By using RBAC, you can closely align the roles you assign users - and administrators to the actual roles they hold within your organization, so that they have - access to just the information they need, and therefore meet compliance requirements. For - more information, see RBAC.

    +

    Couchbase Server 5.5 is a substantial extension of the Couchbase Data Platform. This release + introduces several new enterprise grade features relating to agility, performance, and + manageability. It also includes enhancements to existing services as well as new server and + SDK components that have been in the works for a long time. Take a look at the release notes for a list of + known and fixed issues in this release.

    + +
    Unmatched agility and flexibility +

    Couchbase Eventing Service

    +

    This release introduces our Couchbase Eventing Service – a multidimensional-scaling (MDS) + enabled service that lets you process changes to data as they happen in near real-time.

    +

    Couchbase Functions is the first offering under the Couchbase Eventing Service that enables + you to write server-side functions that are automatically triggered using the familiar + Event-Condition-Action model. It provides an easy-to-use online JavaScript code editor that + lets you develop and debug the code on generated mutations to the data residing in Couchbase + cluster.

    +

    The Couchbase Eventing Service handles all the complexity of scaling to a large number of + document mutations. This new offering enables you to develop deeply engaging and + personalized user experiences based on real-time events. To learn more about this new + service, see .

    +

    Couchbase Analytics (Developer Preview)

    +

    Couchbase Analytics (currently in Developer Preview) enables you to analyze your JSON data + in real time without the need to extract, transform, load (ETL) your data into a separate + system. It allows you to develop insight-driven applications quickly and easily using the + same SDKs. It removes the need for ETL and the complexity of managing a separate analytics + system. With this release, Analytics is fully integrated with Couchbase Server and no longer + requires a separate installation. Couchbase’s unique architecture for Multi-Dimensional + Scaling (MDS) allows operational and analytical queries to run concurrently without + impacting one another, providing the workload isolation required for production deployments. + To learn more about this service, see Couchbase Analytics.

    +

    ANSI joins in N1QL

    +

    The Couchbase Data Platform is the only platform that extends SQL to support JSON data + natively via N1QL queries. N1QL now has support for ANSI joins making it easier to migrate + applications from RDBMS to Couchbase. Developers can leverage their SQL knowledge to develop + applications on Couchbase, thereby increasing productivity.

    +

    ANSI joins support improved data modeling capability and flexibility with Couchbase. Since + you can join based on arbitrarily complex expressions, business requirements can be coded + and translated effectively and efficiently. We've extended ANSI syntax to join on both + scalars and arrays. See the + for details.

    -
    + + +

    + +
    Arguments
    +
    +
    + +
    expr
    +
    [Required] The array of items to be evaluated.
    +     +
    +
    +     +
    + +
    Return Values
    +
    An array with distinct elements of the input array expr.
    +
    If the input expr is MISSING, it returns + MISSING.
    +
    If the input expr is a non-array value, it returns + NULL.
    +     +
    Example 6: Use ARRAY_DISTINCT with a group of items. + SELECT ARRAY_DISTINCT(["apples","bananas","grapes","oranges","apples","mangoes","bananas"]) AS distinct_fruits; Results: @@ -328,10 +372,8 @@ Results: "apples" ] } -] -
    - -
    ARRAY_FLATTEN(<varname>expression</varname>, +]</codeblock></section> + <section id="fn-array-flatten"><title>ARRAY_FLATTEN(<varname>expr</varname>, <varname>depth</varname>)

    Description
    @@ -344,7 +386,7 @@ Results:
    -
    expression
    +
    expr
    [Required] The multilevel array to be flattened.
    @@ -360,10 +402,10 @@ Results:
    Return Value
    An array with depth fewer levels than the input array - expression.
    + expr.
    If one of the arguments is MISSING, it returns MISSING.
    -
    If the input expression is a non-array, or if the input +
    If the input expr is a non-array, or if the input depth argument is not an integer, it returns NULL.
     @@ -410,12 +452,11 @@ Results: ] } ]
    -
    ARRAY_AGG(<varname>expression</varname>) -

    +
    ARRAY_IFNULL(<varname>expr</varname>)

    Description
    -
    This function returns an array of the non-MISSING group values in - the input expression, including NULL values.
    +
    This function parses the input array expr and returns the first + non-NULL value in the array.
    @@ -423,49 +464,7 @@ Results:
    -
    expression
    -
    [Required] The group of elements you wish to output in an array.
    -     -
    -
    -     -
    - -
    Return Values
    -
    An array of non-MISSING values.
    -
    If the input expression is MISSING or if one of - the elements in the array is MISSING, then it returns - MISSING.
    -     -
    Example 8: Use ARRAY_AGG to group a list of three items into an array.

    - SELECT ARRAY_AGG(["abc",1,NULL]) AS array_aggregate; - -Results: -[ - { - "array_aggregate": [ - [ - "abc", - 1, - null - ] - ] - } -] -
    -
    ARRAY_IFNULL(<varname>expression</varname>)

    - -
    Description
    -
    This function parses the input array expression and returns the - first non-NULL value in the array.
    -     -
    - -
    Arguments
    -
    -
    - -
    expression
    +
    expr
    [Required] The array of values to be evaluated.
    @@ -475,9 +474,9 @@ Results:
    Return Values
    The first non-NULL value in the input array.
    -
    If the input expression is MISSING, then it returns +
    If the input expr is MISSING, then it returns MISSING.
    -
    If the input expression is a non-array, then it returns +
    If the input expr is a non-array, then it returns NULL.
    Example 9: Find the first non-NULL value in an array of @@ -505,10 +504,10 @@ Results: "if_null": null } ]
    -
    ARRAY_INSERT(<varname>expression</varname>, <varname>position</varname>, - <varname>value1</varname>, <varname>value2</varname>, …, - <varname>valueX</varname>) -

    +

    + ARRAY_INSERT(<varname>expr</varname>, <varname>pos</varname>, <varname>val1</varname>, + <varname>val2</varname>, …) +

    Description
    @@ -523,23 +522,22 @@ Results:
    -
    expression
    +
    expr
    [Required] The array to insert items into.
    -
    position
    -
    [Required] The integer specifying the array location from the left of the - input array expression, where the 1st position is 0 - (zero).
    +
    pos
    +
    [Required] The integer specifying the array position from the left of the + input array expr, where the 1st position is 0 (zero).
    -
    value1, value2, … , valueX
    -
    [Required] The value or multiple value items to insert into the input array - expression.
    +
    val1, val2, …
    +
    [At least one is Required] The value or multiple value items to insert into + the input array expression.
    @@ -549,10 +547,10 @@ Results:
    Return Values
    An array with the input value or multiple value items inserted into the input array - expression at position position.
    + expression at position pos.
    If any of the three arguments are MISSING, then it returns MISSING.
    -
    If the expression argument is a non-array or if the +
    If the expr argument is a non-array or if the position argument is not an integer, then it returns NULL.
    @@ -580,11 +578,11 @@ Result:[ ] } ]

    -
    - -
    ARRAY_INTERSECT(<varname>expression1</varname>, <varname>expression2</varname>, ...) -

    (Introduced in Couchbase Server 4.5.1)

    -

    +

    +
    + ARRAY_INTERSECT(<varname>expr1</varname>, <varname>expr2</varname>, ...) +

    (Introduced in Couchbase Server 4.5.1)

    +

    Description
    @@ -599,7 +597,7 @@ Result:[
    -
    expression1, expression2, ...
    +
    expr1, expr2, …
    [At least 2 are Required] The two or more arrays to compare the values of.
     @@ -642,9 +640,10 @@ Result: "array_intersection": [] } ]

    -
    -
    ARRAY_LENGTH(<varname>expression</varname>) -

    +

    +
    + ARRAY_LENGTH(<varname>expr</varname>) +

    Description
    @@ -657,7 +656,7 @@ Result:
    -
    expression
    +
    expr
    [Required] The array whose elements you want to know the number of.
    @@ -687,9 +686,10 @@ Result: "total_likes": 8 } ]

    -
    -
    ARRAY_MAX(<varname>expression</varname>) -

    +

    +
    + ARRAY_MAX(<varname>expr</varname>) +

    Description
    @@ -703,7 +703,7 @@ Result:
    -
    expression
    +
    expr
    [Required] The array whose elements you want to know the highest value of.
     @@ -716,9 +716,9 @@ Result:
    Return Values
    The largest non-NULL, non-MISSING array element, in N1QL collation order.
    -
    If the input expression is MISSING, then it - returns MISSING.
    -
    If the input expression is a non-array value, then it returns +
    If the input expr is MISSING, then it returns + MISSING.
    +
    If the input expr is a non-array value, then it returns NULL.
    @@ -735,8 +735,9 @@ Results: "max_val": "Vallie Ryan" } ]

    -
    -
    ARRAY_MIN(<varname>expression</varname>) +
    +
    + ARRAY_MIN(<varname>expr</varname>)

    @@ -751,7 +752,7 @@ Results:
    -
    expression
    +
    expr
    [Required] The array whose elements you want to know the lowest value of.
    @@ -763,9 +764,9 @@ Results:
    Return Values
    The smallest non-NULL, non-MISSING array element, in N1QL collation order.
    -
    If the input expression is MISSING, then it - returns MISSING.
    -
    If the input expression is a non-array value, then it returns +
    If the input expr is MISSING, then it returns + MISSING.
    +
    If the input expr is a non-array value, then it returns NULL.
    @@ -785,9 +786,10 @@ Results: } ]

    -
    -
    ARRAY_POSITION(<varname>expression</varname>, <varname>value</varname>) -

    +

    +
    + ARRAY_POSITION(<varname>expr</varname>, <varname>val</varname>) +

    Description
    @@ -802,13 +804,13 @@ Results:
    -
    expression
    +
    expr
    [Required] The array you want to search through.
    -
    value
    +
    val
    [Required] The value you're searching for and whose position you want to know.
     @@ -819,10 +821,9 @@ Results:
    Return Values
    -
    An integer representing the first position of the input value, +
    An integer representing the first position of the input val, where the first position is 0.
    -
    It returns -1 if the input value does not exist in the - array.
    +
    It returns -1 if the input val does not exist in the array.
    If one of the arguments is MISSING, it returns MISSING.
    If either of the arguments are non-array values, it returns @@ -842,16 +843,17 @@ LIMIT 1; "array_position": 4 } ]

    -
    -
    ARRAY_PREPEND(<varname>value1</varname>, <varname>value2</varname>, ... , - <varname>valueX</varname>, <varname>expression</varname>) -

    +

    +
    + ARRAY_PREPEND(<varname>val1</varname>, <varname>val2</varname>, … , + <varname>expr</varname>) +

    Description
    This function returns the new array after prepending the array - expression with the specified value or - multiple value arguments.
    + expr with the specified val or multiple + val arguments.
    It requires a minimum of two arguments.
    @@ -861,16 +863,16 @@ LIMIT 1;
    -
    value1, value2, … ,valueX
    +
    val1, val2, …
    [At least 1 is Required] The value or multiple value arguments to prepend to - the input expression.
    + the input expr.
    expression
    -
    [Required] The array you want to have the input value or - value arguments prepended to.
    +
    [Required] The array you want to have the input value + argument(s) prepended to.
    @@ -879,8 +881,8 @@ LIMIT 1;
    Return Values
    -
    A new array with the input value or value - arguments prepended to the input array expression.
    +
    A new array with the input val argument(s) prepended to the input + array expr.
    If one of the arguments is MISSING, it returns MISSING.
    If the last argument is a non-array, it returns NULL.
    @@ -909,16 +911,17 @@ Results: ] } ]

    -
    -
    ARRAY_PUT(<varname>expression</varname>, <varname>value1</varname>, - <varname>value2</varname>, … , <varname>valueX</varname>) -

    +
    +
    + ARRAY_PUT(<varname>expr</varname>, <varname>val1</varname>, <varname>val2</varname>, + …) +

    Description
    -
    This function returns a new array with value or multiple - value arguments appended if the value is not +
    This function returns a new array with val or multiple + val arguments appended if the val is not already present. Otherwise, it returns the unmodified input array - expression.
    + expr.
    It requires a minimum of two arguments.
    @@ -927,14 +930,14 @@ Results:
    -
    expression
    +
    expr
    [Required] The array you want to append the input value or value arguments.
    -
    value1, value2, … , valueX
    +
    val1, val2, …
    [At least 1 is Required] The value or multiple value arguments that you want appended to the end of the input array expression.
     @@ -944,9 +947,9 @@ Results:
    Return Values
    -
    A new array with value or multiple value - arguments appended if the value is not already present. Otherwise, - it returns the unmodified input array expression.
    +
    A new array with val or multiple val arguments + appended if the val is not already present. Otherwise, it returns + the unmodified input array expr.
    If one of the arguments is MISSING, then it returns MISSING.
    If the first argument is a non-array, then it returns NULL.
    @@ -976,8 +979,8 @@ Results: } ]

    -
    -
    ARRAY_RANGE(<varname>start_num</varname>, + </section> + <section id="fn-array-range"><title>ARRAY_RANGE(<varname>start_num</varname>, <varname>end_num</varname> [, <varname>step_num</varname> ])

    @@ -1068,50 +1071,48 @@ Results: ] } ]
    -
    ARRAY_REMOVE(<varname>expression</varname>, <varname>value1</varname>, - <varname>value2</varname>, …, <varname>valueX</varname>) -
    - -
    Description
    -
    This function returns a new array with all occurrences of the specified - value or multiple value fields removed from - the array expressionand it requires a minimum of two arguments. -
    -     -
    -
    - -
    Arguments
    -
    -
    - -
    expression
    -
    [Required] The input array to have the specified value or - multiple value fields removed.
    -     -
    -
    - -
    value1, value2, … , valueX
    -
    [At least 1 is Required] The input value or multiple values to remove from the - input array expression.
    -     -
    -
    -     -
    -
    - -
    Output Values
    -
    A new array with all occurrences of the specified value or - multiple value fields removed from the array - expression.
    -
    If any of the arguments are MISSING, then it returns - MISSING.
    -
    If the first argument is not an array, then it returns NULL.
    -     -
    - +
    + ARRAY_REMOVE(<varname>expr</varname>, <varname>val1</varname>, <varname>val2</varname>, + …) +
    + +
    Description
    +
    This function returns a new array with all occurrences of the specified + value or multiple value fields removed from the + array expressionand it requires a minimum of two arguments.
    +     +
    +
    + +
    Arguments
    +
    +
    + +
    expr
    +
    [Required] The input array to have the specified val or + multiple val fields removed.
    +     +
    +
    + +
    val1, val2, …
    +
    [At least 1 is Required] The input value or multiple values to remove from the + input array expr.
    +     +
    +
    +     +
    +
    + +
    Output Values
    +
    A new array with all occurrences of the specified val or multiple + val fields removed from the array expr.
    +
    If any of the arguments are MISSING, then it returns + MISSING.
    +
    If the first argument is not an array, then it returns NULL.
    +     +

    Example 21: Remove "Vallie Ryan" from the public_likes array.

    SELECT ARRAY_REMOVE(t.public_likes, "Vallie Ryan") AS remove_val @@ -1134,14 +1135,15 @@ Results: } ]

    -
    -
    ARRAY_REPEAT(<varname>value</varname>, <varname>rep_int</varname>) -

    +

    +
    + ARRAY_REPEAT(<varname>val</varname>, <varname>rep_int</varname>) +

    Description
    -
    This function returns a new array with the specified value - repeated rep_int times.
    +
    This function returns a new array with the specified val repeated + rep_int times.
    @@ -1150,7 +1152,7 @@ Results:
    -
    value
    +
    val
    [Required] The input value you want repeated.
    @@ -1158,7 +1160,7 @@ Results:
    rep_int
    [Required] The integer number of times you want the input - value repeated.
    + val repeated.
    @@ -1167,7 +1169,7 @@ Results:
    Output Values
    -
    A new array with the specified value repeated +
    A new array with the specified val repeated rep_int times.
    If any of the arguments are MISSING, then it returns MISSING.
    @@ -1176,7 +1178,8 @@ Results:

    -

    Example 22: Make an array with "Vallie Ryan" three times.

    SELECT ARRAY_REPEAT("Vallie Ryan", 3) AS repeat_val; +

    Example 22: Make an array with "Vallie Ryan" three times.

    + SELECT ARRAY_REPEAT("Vallie Ryan", 3) AS repeat_val; Results: [ @@ -1188,64 +1191,64 @@ Results: ] } ] -
    -
    ARRAY_REPLACE(<varname>expression</varname>, <varname>value1</varname>, - <varname>value2</varname> [, <varname>max_int</varname> ]) -
    - -
    Description
    -
    This function returns a new array with all occurrences of - value1 replaced with value2.
    -
    If max_int is specified, than no more than - max_int replacements will be performed.
    -     -
    +
    +
    ARRAY_REPLACE(<varname>expr</varname>, + <varname>val1</varname>, <varname>val2</varname> [, <varname>max_int</varname> ])
    - -
    Arguments
    -
    -
    - -
    expression
    -
    [Required] The input array you want to replace value1 with - value2.
    -     -
    -
    - -
    value1
    -
    [Required] The existing value in the input expression you - want to replace.
    -     -
    -
    - -
    value2
    -
    [Required] The new value you want to take the place of - value1 in the input expression.
    -     -
    -
    - -
    max_int
    -
    [Optional. Default is no maximum] The number of maximum replacements to - perform.
    -     -
    -
    -     -
    - -
    Return Values
    -
    A new array with all or max_int occurrences of - value1 replaced with value2.
    -
    If any of the arguments are MISSING, then it returns - MISSING.
    -
    If the first argument is not an array or if the second argument is - NULL, then it returns NULL.
    -     -
    Example 23: Replace all occurrences of "Vallie Ryan" with "Valerie Ryan". - SELECT ARRAY_REPLACE(t.public_likes, "Vallie Ryan", "Valerie Ryan") AS replace_val + +
    Description
    +
    This function returns a new array with all occurrences of + value1 replaced with value2.
    +
    If max_int is specified, than no more than + max_int replacements will be performed.
    +     + +
    + +
    Arguments
    +
    +
    + +
    expr
    +
    [Required] The input array you want to replace val1 with + val2.
    +     +
    +
    + +
    val1
    +
    [Required] The existing value in the input expr you want to + replace.
    +     +
    +
    + +
    val2
    +
    [Required] The new value you want to take the place of val1 + in the input expr.
    +     +
    +
    + +
    max_int
    +
    [Optional. Default is no maximum] The number of maximum replacements to + perform.
    +     +
    +
    +     +
    + +
    Return Values
    +
    A new array with all or max_int occurrences of + val1 replaced with val2.
    +
    If any of the arguments are MISSING, then it returns + MISSING.
    +
    If the first argument is not an array or if the second argument is + NULL, then it returns NULL.
    +     +
    Example 23: Replace all occurrences of "Vallie Ryan" with "Valerie Ryan". + SELECT ARRAY_REPLACE(t.public_likes, "Vallie Ryan", "Valerie Ryan") AS replace_val FROM `travel-sample` t WHERE type="hotel" LIMIT 1; @@ -1265,39 +1268,38 @@ Results: ] } ] - -
    -
    ARRAY_REVERSE(<varname>expression</varname>) -
    - -
    Description
    -
    This function returns a new array with all the elements of - expression in reverse order.
    -     -
    -
    - -
    Arguments
    -
    -
    - -
    expression
    -
    [Required] The input array whose elements you want to reverse.
    -     -
    -
    -     -
    -
    - -
    Return Values
    -
    A new array with all the elements of expression in reverse - order.
    -
    If the argument is MISSING, then it returns - MISSING.
    -
    If the argument is a non-array value, then it returns NULL.
    -     -
    +
    +
    + ARRAY_REVERSE(<varname>expr</varname>) +
    + +
    Description
    +
    This function returns a new array with all the elements of expr in + reverse order.
    +     +
    +
    + +
    Arguments
    +
    +
    + +
    expr
    +
    [Required] The input array whose elements you want to reverse.
    +     +
    +
    +     +
    +
    + +
    Return Values
    +
    A new array with all the elements of expr in reverse order.
    +
    If the argument is MISSING, then it returns + MISSING.
    +
    If the argument is a non-array value, then it returns NULL.
    +     +

    Example 24: Reverse the values in the public_likes array.

    SELECT ARRAY_REVERSE(t.public_likes) AS reverse_val @@ -1321,13 +1323,14 @@ Results: } ]

    -
    -
    ARRAY_SORT(<varname>expression</varname>) -

    +

    +
    + ARRAY_SORT(<varname>expr</varname>) +

    Description
    -
    This function returns a new array with the elements of expression +
    This function returns a new array with the elements of expr sorted in N1QL collation order.
    @@ -1337,7 +1340,7 @@ Results:
    -
    expression
    +
    expr
    [Required] The input array you want sorted.
    @@ -1347,8 +1350,8 @@ Results:
    Return Values
    -
    A new array with the elements of expression sorted in N1QL - collation order.
    +
    A new array with the elements of expr sorted in N1QL collation + order.
    If the argument is MISSING, then it returns MISSING.
    If the argument is a non-array value, then it returns NULL.
    @@ -1378,14 +1381,15 @@ Results: } ]

    -
    -
    ARRAY_STAR(<varname>expression</varname>) -

    +

    +
    + ARRAY_STAR(<varname>expr</varname>) +

    Description
    -
    This function converts an array of expression objects into an - object of arrays.
    +
    This function converts an array of expr objects into an object of + arrays.
    @@ -1394,7 +1398,7 @@ Results:
    -
    expression
    +
    expr
    [Required] The input array you want to convert into an object of arrays.
    @@ -1413,8 +1417,8 @@ Results:

    Example 26: Convert a given array of two documents each with five items into an object of five arrays each with two documents.

    -

    - SELECT ARRAY_STAR( [ +

    + SELECT ARRAY_STAR( [ { "address": "Capstone Road, ME7 3JE", "city": "Medway", @@ -1457,20 +1461,22 @@ Results: } } ] -

    -

    Array references ( doc.f[*].id )

    -

    You can use an asterisk (*) as an array subscript which converts the array to an object of - arrays. The following example returns an array of the ages of the given contact’s children: - SELECT children[*].age FROM contacts WHERE fname = "Dave"

    -

    An equivalent query can be written using the array_star() function: - SELECT array_star(children).age FROM contacts WHERE fname = "Dave"

    -
    ARRAY_SUM(<varname>expression</varname>) -

    +

    +

    Array references ( doc.f[*].id )

    +

    You can use an asterisk (*) as an array subscript which converts the array to an object of + arrays. The following example returns an array of the ages of the given contact’s children: + SELECT children[*].age FROM contacts WHERE fname = "Dave"

    +

    An equivalent query can be written using the array_star() function: + SELECT array_star(children).age FROM contacts WHERE fname = "Dave"

    +
    +
    + ARRAY_SUM(<varname>expr</varname>) +

    Description
    This function returns the sum of all the non-NULL number values in - the expression array.
    + the expr array.
    @@ -1479,7 +1485,7 @@ Results:
    -
    expression
    +
    expr
    [Required] The input array of numbers you want to know the total value of.
     @@ -1491,7 +1497,7 @@ Results:
    Return Values
    The sum of all the non-NULL number values in the - expression array.
    + expr array.
    If there are no number values, then it returns 0 (zero).
    If the argument is MISSING, then it returns MISSING.
    @@ -1500,7 +1506,7 @@ Results:

    Example 27: Find the total of a given array of numbers.

    - SELECT ARRAY_SUM([0,1,1,2,3,5]) as sum; + SELECT ARRAY_SUM([0,1,1,2,3,5]) as sum; Results: [ @@ -1508,10 +1514,10 @@ Results: "sum": 12 } ] -
    -
    ARRAY_SYMDIFF(<varname>expression1</varname>, - <varname>expression2</varname>, …, <varname>expressionX</varname>)

    Synonym: - ARRAY_SYMDIFF1(expression1, expression2, … , expressionX)

    +

    +
    ARRAY_SYMDIFF(<varname>expr1</varname>, + <varname>expr2</varname>, …)

    Synonym: ARRAY_SYMDIFF1(expression1, + expression2, …)

    Description
    @@ -1527,7 +1533,7 @@ Results:
    -
    expression1, expression2, … , expressionX
    +
    expr1, expr2, …
    [At least 2 are Required] The input arrays to compare.
    @@ -1565,8 +1571,8 @@ Results: ] } ]
    -
    ARRAY_SYMDIFFN(<varname>expression1</varname>, - <varname>expression2</varname>, … , <varname>expressionX</varname>)

    +

    ARRAY_SYMDIFFN(<varname>expr1</varname>, + <varname>expr2</varname>, …)

    Description
    @@ -1582,7 +1588,7 @@ Results:
    -
    expression1, expression2, … , expressionX
    +
    expr1, expr2, …
    [At least 2 are Required] The input arrays to compare.
    @@ -1621,8 +1627,8 @@ Results: ] } ]
    -
    ARRAY_UNION(<varname>expression1</varname>, - <varname>expression2</varname>, … , <varname>expressionX</varname>)

    +

    ARRAY_UNION(<varname>expr1</varname>, + <varname>expr2</varname>, …)

    Description
    @@ -1636,7 +1642,7 @@ Results:
    -
    expression1, expression2, … , expressionX
    +
    expr1, expr2, …
    [At least 2 are Required] The input arrays to compare.
    @@ -1676,5 +1682,5 @@ Results: "array_union": null } ]
    - + diff --git a/content/n1ql/n1ql-language-reference/createindex.dita b/content/n1ql/n1ql-language-reference/createindex.dita index d61956a12e..13bc75b101 100644 --- a/content/n1ql/n1ql-language-reference/createindex.dita +++ b/content/n1ql/n1ql-language-reference/createindex.dita @@ -18,6 +18,13 @@

    Both GSI and View indexes provide a status field and mark index status pending. With GSI indexer, index status continues to report "pending". This status field and other index metadata can be queried using system:indexes.

    +

    You can create multiple identical secondary indexes on a bucket for better index availability and place them on separate nodes using the "nodes" clause.

    +

    Array indexing enables you to create global indexes on array elements and optimize the + execution of queries involving array elements. For details, see .

    +

    Index partitioning helps increase the query performance by dividing and spreading a large + index of documents across multiple nodes, horizontally scaling out an index as needed. + For details, see .

    RBAC Privileges

    User executing the CREATE INDEX statement must have the Query Manage Index privilege granted on the keyspace/bucket. For more details about user roles, see "errors": [{"code": 12014, "msg": "error: Build Already In Progress. Bucket BUCKET_NAME. Index INDEX_NAME. Index state: pending"}] -

    You can create multiple identical secondary indexes on a bucket for better index - availability and place them on separate nodes using the "nodes" clause.

    -

    - Couchbase Server 4.5 release adds the capability to create global - indexes on array elements and optimizes the execution of queries involving array - elements. See for details. -

    Syntax

    CREATE INDEX index_name ON named_keyspace_ref ( expression1 [ , expression2 ] * ) WHERE filter_expressions diff --git a/content/n1ql/n1ql-language-reference/createprimaryindex.dita b/content/n1ql/n1ql-language-reference/createprimaryindex.dita index 8a89fc5626..ec3bba8d78 100644 --- a/content/n1ql/n1ql-language-reference/createprimaryindex.dita +++ b/content/n1ql/n1ql-language-reference/createprimaryindex.dita @@ -18,7 +18,6 @@

    Both GSI and view indexers provide a status field and mark index status pending. With the GSI indexer, index status continues to report pending. This status field and other index metadata can be queried by using system:indexes.

    -

    RBAC Privileges

    Users executing the CREATE PRIMARY INDEX statement must have the Query Manage Index privilege granted on the keyspace. For more details about user @@ -40,7 +39,7 @@ CREATE PRIMARY INDEX [index_name] ON named_keyspace_ref [ USING GSI | USING VIEW ] - [ WITH {"nodes": ["node_name"], "defer_build":true|false} ]; + [ WITH {"nodes": ["node_name"], "defer_build":true|false}, "num_replica": num_replica_num } ];

    @@ -93,8 +92,8 @@
    WITH options
    -
    The WITH clause specifies additional options - for the GSI type primary indexes.
    +
    The WITH clause specifies additional options for the + GSI type primary indexes.
    "nodes":["node name"]
    A single primary index of type GSI @@ -103,7 +102,20 @@ allows you to specify the node that the index is placed on. If nodes is not specified, one of the nodes running the index - service is randomly picked for the index.
    + service is randomly selected to host the index.
    +
    Multiple nodes can be specified to distribute + replicas of an index amongst multiple nodes.

    If + specifying both nodes and + num_replica, the number of + nodes in the array must be one greater than the + specified number of replicas otherwise the index + creation will fail.

    The + node names passed to the nodes + parameter must include the cluster administration + port (by default 8091). For example WITH + {"nodes": ["192.0.2.0:8091"]} + instead of WITH {"nodes": + ["192.0.2.0"]}.
    @@ -127,6 +139,28 @@ building of the index of type GSI.
     +
    + +
    "num_replica": num_replica_num
    +
    num_replica specifies the number + of replicas of the index to create.
    +
    The indexer will automatically distribute these + indexes amongst index nodes in the cluster for + load-balancing and high availability purposes. When + creating an index with replicas in this manner, the + indexer will attempt to distribute the replicas + based on the server groups in use in the cluster + where possible.
    +
    If num_replica_num is not less + than the number of index nodes in the cluster, then + the index creation will fail.
    +
    + We recommend that you do not + create (or drop) secondary indexes when any node + with a secondary index role is down as this may + result in duplicate index names. +
    +
    diff --git a/content/n1ql/n1ql-language-reference/curl.dita b/content/n1ql/n1ql-language-reference/curl.dita index 65330729b5..f89438a3f0 100644 --- a/content/n1ql/n1ql-language-reference/curl.dita +++ b/content/n1ql/n1ql-language-reference/curl.dita @@ -99,8 +99,8 @@ Transfer-related Options - - + + Option @@ -242,22 +242,35 @@

    Creating a Whitelist for CURL() Access

    A whitelist allows a Full Administrator to list out the permitted REST endpoints and URLs for the CURL() function. To enable access based on the whitelist, a Full - Administrator must create the file containing the whitelist, - curl_whitelist.json with the all_access field set to false, - and save it in the same directory as that of the N1QL - certificates(.../Couchbase/var/lib/couchbase/n1qlcerts). Note that - this file is available to only someone with access to localhost, such - as the Full Administrator, and is not visible to the user.

    Each query node in the - system needs to define a curl_whitelist.json file.

    The CURL() - function can access URLs that satisfy a prefix match, which means only URLs on the list (or - prefixes) specified in the curl_whitelist.json file.

    The - whitelist file is specified as a JSON object with the fields described in the following - table. + Administrator must create the file containing the whitelist, which can be created two + ways:
      +
    1. From the Query Workbench UI in the Settings > Advanced Query Settings + section.
      2. +
    2. From CBQ via a cURL command.
      3. +
    1. From the Query Workbench

    In the Query Workbench, navigate to the + Settings > Advanced Query Settings section as shown below:

    After expanding + the Advanced Query Settings section, you can choose the Function Access:
      +
    • Restricted - Access applies only the sites explicitly listed.
      • +
    • Unrestricted - Access applies to all sites within the explicitly listed + sites.
      • +
    Under the Allowed CURL URLs and Disallowed CURL URLs headings, enter your + allowed or disallowed URL in the appropriate textbox and press the Enter key or click + anywhere else on this screen to enter your URL.

    Click to add another URL + to the list.

    Click to remove a URL from the list.

    2. From + CBQ

    From a CBQ prompt, you can send a CURL() command to allow or disallow specific + URLs, for + example:curl -X POST -u Administrator:password \ -d '{"all_access": true, "allowed_urls" : ["company1.com", "couchbase.com"], "disallowed_urls" : ["company2.com"] }' http://localhost:9000/settings/querySettings/curlWhitelist

    The + whitelist file command structure is described in the following table.

    Structure of Whitelist for CURL() - + @@ -273,7 +286,7 @@ boolean This field defines whether the user has access to all URLs or only URLs specified in the allowed_urls array.

    The - curl_whitelist.json file must contain the + curl command must contain the all_access field set to false to enable whitelisting and restrict access.

    Setting this field to true enables access to all endpoints.

    @@ -305,117 +318,7 @@     -

    The following tables lists the curl_whitelist.json JSON object for different - scenarios and CURL whitelist is enabled and

    - - - - - - - curl_whitelist.json - Example - - - - - CURL is disabled when the curl_whitelist.json does not exist. - CURL is disabled. - - - To allow access to all endpoints, the all_access field - is set to - true.{ - "all_access":true, - "allowed_urls":[], - "disallowed_urls":[] -}{ - "all_access":true, - "allowed_urls":["https://maps.googleapis.com/maps/api/geocode/json"] -} - - All CURL queries can run. - - - To restrict access to values specified in allowed_urls, - the all_access field is set to false and - allowed_urls field contains the URLs that can be accessed - by - CURL.{ - "all_access":false, - "allowed_urls":["https://maps.googleapis.com/maps/api/geocode/json"] -}

    If - the allowed_urls field is changed to contain - https://maps.googleapis.com, all Google API queries and - endpoints are accessible by CURL. - { - "all_access":false, - "allowed_urls":["https://maps.googleapis.com"] -}

     - Only queries with the specified URL can be run. All other queried return an - error with the message "Cannot access <url>. Access - denied."SELECT -CURL("https://maps.googleapis.com/maps/api/geocode/json", - { - "data":["address=santa+cruz","components=country:ES"], - "get":true - } -) -     - - If both allowed_urls and - disallowed_urls fields are populated, - disallowed_urls takes precedence over - allowed_urls. - { - "All_access":false, - "allowed_urls":["https://maps.googleapis.com/maps/api/geocode/json"], - "disallowed_urls":["https://maps.googleapis.com"] -} - Since disallowed_urls takes precedence over - allowed_urls, it means all Google API queries and endpoints - are inaccessible. - - - -
    -

    - - CURL Whitelisting Errors - - - - - - Error Condition - Error Message - - - - - curl_whitelist.json does not exist. - n1qlcerts directory does not exist under - .../var/lib/couchbase/ -

    CURL() functionality is disabled. File - curl_whitelist.json does not exist under - ../Couchbase/var/lib/couchbase/n1qlcerts/ directory. - - - - curl_whitelist.json is empty, or the fields are empty, - or all_access field does not exist. - Access denied to CURL(). File curl_whitelist.json is - empty. - - - curl_whitelist.json is an invalid JSON object. - Access denied to CURL(). File curl_whitelist.json - contains invalid JSON. Contents have to be a JSON object. - -

    - -
    -

    +

    Design Considerations

    Here are some of the design considerations to keep in mind when using CURL:

    @@ -436,14 +339,43 @@ FROM bucket b; refers to the alias b and the attribute url in the document. In the second argument, the string concatenation operator (||) cannot be evaluated.

    • -
  • When using CURL whitelisting, the

    • Examples +
      +
    • Ex 1: Use Google Maps API to convert static + address into coordinates.
      • +
    • Ex 2: Use Google Maps API to extract geometry + (address and geographic location bounds) of a given street address.
      • +
    • Ex 3: Join two buckets on different Couchbase + clusters.
      • +
    • Ex 4: FTS Fuzzy Search in a N1QL + query.
      • +
    • Ex 5: Use Yahoo Finance API in a WHERE clause + to find a stock's lowest value for the day.
      • +
    • Ex 6: Use CURL() to allow two URLs and + disallow one URL.
      • +
    • Ex 7: Use CURL() to allow all access to all + endpoints.
      • +
    • Ex 8: Use CURL() to turn off access to all + endpoints and clear the Allowed and Disallowed lists.
      • +
    • Ex 9: Use CURL() to turn off access to all + endpoints but make no changes to the Allowed and Disallowed lists.
      • +
    • Ex 10: Use CURL() to turn off access to all + endpoints, allow one URL, and clear the Disallowed list.
      • +
    • Ex 11: Use CURL() to turn off access to all + endpoints, disallow one URL, and clear the Allowed list.
      • +
    • Ex 12: Use CURL() to allow an IP address and + port instead of a website name.
      • +
    • Ex 13: Use CURL() to allow and disallow the + same URL--and get an error.
      • +
    • Ex 14: Use CURL() with dynamic named + parameters
      • +

    The following examples are using CURL in the query projection list.

    -

    Example 1: The following N1QL query fetches details about the address "Half Moon +

    Example 1: The following N1QL query fetches details about the address "Half Moon Bay" using the Google maps API. The Geocoding API from Google Maps allows you to convert static addresses into coordinates. (For more information refer to

    -

    Example 2: This is similar to Example 1, but following N1QL query fetches details +

    Example 2: This is similar to Example 1, but following N1QL query fetches details about Santa Cruz in Spain using Google geocoding API and extracts the ‘geometry’ field from the result. This query retrieves the address and geographic location bounds of @@ -632,7 +564,7 @@ Results: } ]

    -

    Example 3: This N1QL query shows how to JOIN two buckets on different Couchbase +

    Example 3: This N1QL query shows how to JOIN two buckets on different Couchbase clusters. It is same as explained in the JOIN Clause example, but with the left and right side buckets for the JOIN are in two different Couchbase clusters.

    @@ -701,7 +633,7 @@ LIMIT 4; }, … ]

    -

    Example 4: The following example shows how to use fuzzy search from the FTS service +

    Example 4: The following example shows how to use fuzzy search from the FTS service in a N1QL query. Assuming the FTS index fts_travel is created on the bucket `travel-sample`, running the following N1QL query finds all documents that have "sanfrancisco" anywhere in the document, using the full-text searching @@ -732,7 +664,7 @@ Results: ] } ] -

    Example 5: The following example uses the CURL() function with a +

    Example 5: The following example uses the CURL() function with a WHERE clause. It uses the Yahoo finance API to find the day’s low value (i.e DaysLow) of HDP stock and finds all the documents in the default bucket that have ‘min_threshold’ attribute value greater than the @@ -765,21 +697,70 @@ Results: } ]

    -

    Example 6: The following example uses CURL() whitelisting to allow access to all - Google Maps API and endpoints except - https://maps.googleapis.com/maps/api/geocode/json. - { - "All_access":false, - "allowed_urls":["https://maps.googleapis.com"], - "disallowed_urls":["https://maps.googleapis.com/maps/api/geocode/json"] -} -

    -

    The following query will return an error: "Cannot access <url>. Access - denied".SELECT CURL("https://maps.googleapis.com/maps/api/geocode/json", - { "data":["address=santa+cruz","components=country:ES"], - "get":true - } -)

    +

    Example 6: Use CURL() to allow two URLs and disallow one URL. + curl -X POST -u Administrator:password \ +-d '{ +"all_access": true, +"allowed_urls" : ["company1.com", "couchbase.com"], +"disallowed_urls" : [“company2.com”] +}' http://localhost:9000/settings/querySettings/curlWhitelist

    +

    Example 7: Use CURL() to allow access to all + endpoints.curl -X POST -u Administrator:password \ +-d '{ +"all_access": true +}' http://localhost:9000/settings/querySettings/curlWhitelistExample 8: Use CURL() to turn off access to all endpoints and clear the + Allowed and Disallowed + lists.curl -X POST -u Administrator:password \ +-d '{ +"all_access": false, +"allowed_urls" : [], +"disallowed_urls" : [] +}' http://localhost:9000/settings/querySettings/curlWhitelistExample 9: Use CURL() to turn off access to all endpoints but make no changes + to the Allowed and Disallowed + lists.curl -X POST -u Administrator:password \ +-d '{ +"all_access": false +}' http://localhost:9000/settings/querySettings/curlWhitelistExample 10: Use CURL() to turn off access to all endpoints, allow one URL, + and clear the Disallowed + list.curl -X POST -u Administrator:password \ +-d '{ +"all_access": false, +"allowed_urls" : [“https://maps.googleapis.com/maps/api/geocode/json”], +"disallowed_urls" : [] +}' http://localhost:9000/settings/querySettings/curlWhitelistExample 11: Use CURL() to turn off access to all endpoints, disallow one + URL, and clear the Allowed + list.curl -X POST -u Administrator:password \ +-d '{ +"all_access": false, +"disallowed_urls" : [“https://maps.googleapis.com/maps/api/geocode/json”], +"allowed_urls" : [] +}' http://localhost:9000/settings/querySettings/curlWhitelistExample 12: Use CURL() to allow an IP address and port instead of a website + name.curl -X POST -u Administrator:password \ +-d '{ +"all_access": false, +“disallowed_urls" : [“https://maps.googleapis.com/maps/api/geocode/json”], +"allowed_urls" : [“http://127.0.0.1:9499/query/service”] +}' http://localhost:9000/settings/querySettings/curlWhitelistExample 13: Use CURL() to allow and disallow the same URL--and get an + error.curl -X POST -u Administrator:password \ +-d '{ +"all_access": false, +“disallowed_urls" : [“https://maps.googleapis.com/maps/api/geocode/json”], +"allowed_urls" : [“https://maps.googleapis.com/maps/api/geocode/json”] +}' http://localhost:9000/settings/querySettings/curlWhitelistExample 14: Use CURL() with dynamic named + parameters.SELECT CURL(b.url, $params) FROM bucket b WHERE b.username = “joe”; If + we wanted to use Node.JS, we would + use:bucket.query(SELECT CURL(b.url, $params) FROM bucket b WHERE b.username = “joe”, { params: { data: "..." } }, + (error, result) => {} );$params + is a named parameter, so we name it in the parameters object when executing the + query. Then we populate the properties with the data that's in the documents since those + properties can be variables.

    diff --git a/content/n1ql/n1ql-language-reference/datefun.dita b/content/n1ql/n1ql-language-reference/datefun.dita index 059498fd4b..467a7d3919 100644 --- a/content/n1ql/n1ql-language-reference/datefun.dita +++ b/content/n1ql/n1ql-language-reference/datefun.dita @@ -178,8 +178,8 @@ id="table_z31_3tq_5y"> Date String Examples - - + + Format @@ -546,17 +546,13 @@
    Examples
    SELECT CLOCK_LOCAL() as full_date, CLOCK_LOCAL('invalid date') as invalid_date, - CLOCK_LOCAL('1111-11-11') as short_date; - -{ - "results": [ - { - "full_date": "2017-01-31T03:49:30.612-08:00", - "invalid_date": "2017-01-31T03:49:30.612-08:00", - "short_date": "2017-01-31" - } - ] -}
    + CLOCK_LOCAL('1111-11-11') as short_date;Results:[ + { + "full_date": "2018-01-23T13:57:29.847-08:00", + "invalid_date": "2018-01-23T13:57:29.847-08:00", + "short_date": "2018-01-23" + } +]
    @@ -591,17 +587,11 @@
    Examples
    -
    - SELECT CLOCK_MILLIS(); - -{ - "results": [ - { - "$1": 1483543192439.568 - } - ] -} -
    +
    SELECT CLOCK_MILLIS() AS CurrentTime;Results:[ + { + "CurrentTime": 1516744600430.677 + } +]
    @@ -647,17 +637,13 @@
    Examples
    SELECT CLOCK_STR() as full_date, CLOCK_STR('invalid date') as invalid_date, - CLOCK_STR('1111-11-11') as short_date; - -{ - "results": [ - { - "full_date": "2017-01-31T03:53:52.348-08:00", - "invalid_date": "2017-01-31T03:53:52.348-08:00", - "short_date": "2017-01-31" - } - ] -}
    + CLOCK_STR('1111-11-11') as short_date;Results:[ + { + "full_date": "2018-01-23T13:55:10.798-08:00", + "invalid_date": "2018-01-23T13:55:10.798-08:00", + "short_date": "2018-01-23" + } +]
    @@ -716,19 +702,15 @@ CLOCK_TZ('UTC', '1111-11-11') as UTC_short_date, CLOCK_TZ('invalid timezone') as invalid_timezone, CLOCK_TZ('US/Eastern') as us_east, - CLOCK_TZ('US/Pacific') as us_west; - -{ - "results": [ - { - "UTC_full_date": "2017-01-31T12:31:44.133Z", - "UTC_short_date": "2017-01-31", - "invalid_timezone": null, - "us_east": "2017-01-31T07:31:44.133-05:00", - "us_west": "2017-01-31T04:31:44.133-08:00" - } - ] -} + CLOCK_TZ('US/Pacific') as us_west;Results:[ + { + "UTC_full_date": "2018-01-23T21:54:37.178Z", + "UTC_short_date": "2018-01-23", + "invalid_timezone": null, + "us_east": "2018-01-23T16:54:37.18-05:00", + "us_west": "2018-01-23T13:54:37.181-08:00" + } +]
    @@ -774,16 +756,12 @@
    Examples
    -
    SELECT CLOCK_UTC() as full_date, CLOCK_UTC('1111-11-11') as short_date; - -{ - "results": [ - { - "full_date": "2017-01-31T14:13:03.147Z", - "short_date": "2017-01-31" - } - ] -}
    +
    SELECT CLOCK_UTC() as full_date, CLOCK_UTC('1111-11-11') as short_date;Results:[ + { + "full_date": "2018-01-23T21:54:03.593Z", + "short_date": "2018-01-23" + } +]
    @@ -840,18 +818,14 @@
    SELECT DATE_ADD_MILLIS(1463284740000, 3, 'day') as add_3_days, DATE_ADD_MILLIS(1463284740000, 3, 'year') as add_3_years, DATE_ADD_MILLIS(1463284740000, -3, 'day') as sub_3_days, - DATE_ADD_MILLIS(1463284740000, -3, 'year') as sub_3_years; - -{ - "results": [ - { - "add_3_days": 1463543940000, - "add_3_years": 1557892740000, - "sub_3_days": 1463025540000, - "sub_3_years": 1368590340000 - } - ] -}
    + DATE_ADD_MILLIS(1463284740000, -3, 'year') as sub_3_years;Results:[ + { + "add_3_days": 1463543940000, + "add_3_years": 1557892740000, + "sub_3_days": 1463025540000, + "sub_3_years": 1368590340000 + } +] @@ -906,18 +880,14 @@
    SELECT DATE_ADD_STR('2016-05-15 03:59:00Z', 3, 'day') as add_3_days, DATE_ADD_STR('2016-05-15 03:59:00Z', 3, 'year') as add_3_years, DATE_ADD_STR('2016-05-15 03:59:00Z', -3, 'day') as sub_3_days, - DATE_ADD_STR('2016-05-15 03:59:00Z', -3, 'year') as sub_3_years; - -{ - "results": [ - { - "add_3_days": "2016-05-18T03:59:00Z", - "add_3_years": "2019-05-15T03:59:00Z", - "sub_3_days": "2016-05-12T03:59:00Z", - "sub_3_years": "2013-05-15T03:59:00Z" - } - ] -}
    + DATE_ADD_STR('2016-05-15 03:59:00Z', -3, 'year') as sub_3_years;Results:[ + { + "add_3_days": "2016-05-18T03:59:00Z", + "add_3_years": "2019-05-15T03:59:00Z", + "sub_3_days": "2016-05-12T03:59:00Z", + "sub_3_years": "2013-05-15T03:59:00Z" + } +] @@ -974,18 +944,14 @@
    SELECT DATE_DIFF_MILLIS(1463543940000, 1463284740000, 'day') as add_3_days, DATE_DIFF_MILLIS(1557892740000, 1463284740000, 'year') as add_3_years, DATE_DIFF_MILLIS(1463025540000, 1463284740000, 'day') as sub_3_days, - DATE_DIFF_MILLIS(1368590340000, 1463284740000, 'year') as sub_3_years; - -{ - "results": [ - { - "add_3_days": 3, - "add_3_years": 3, - "sub_3_days": -3, - "sub_3_years": -3 - } - ] -}
    + DATE_DIFF_MILLIS(1368590340000, 1463284740000, 'year') as sub_3_years;Results:[ + { + "add_3_days": 3, + "add_3_years": 3, + "sub_3_days": -3, + "sub_3_years": -3 + } +] @@ -1040,21 +1006,43 @@
    Examples
    -
    SELECT DATE_DIFF_STR('2016-05-18T03:59:00Z', '2016-05-15 03:59:00Z', 'day') as add_3_days, +
    Example 1: Find the day difference and year difference between two + strings.SELECT DATE_DIFF_STR('2016-05-18T03:59:00Z', '2016-05-15 03:59:00Z', 'day') as add_3_days, DATE_DIFF_STR('2019-05-15T03:59:00Z', '2016-05-15 03:59:00Z', 'year') as add_3_years, DATE_DIFF_STR('2016-05-12T03:59:00Z', '2016-05-15 03:59:00Z', 'day') as sub_3_days, - DATE_DIFF_STR('2013-05-15T03:59:00Z', '2016-05-15 03:59:00Z', 'year') as sub_3_years; - -{ - "results": [ - { - "add_3_days": 3, - "add_3_years": 3, - "sub_3_days": -3, - "sub_3_years": -3 - } - ] -}
    + DATE_DIFF_STR('2013-05-15T03:59:00Z', '2016-05-15 03:59:00Z', 'year') as sub_3_years;Results:[ + { + "add_3_days": 3, + "add_3_years": 3, + "sub_3_days": -3, + "sub_3_years": -3 + } +]Example + 2: List all hotel documents that were reviewed between two + dates.SELECT name, reviews[0].date +FROM `travel-sample` +WHERE type = "hotel" +AND reviews[0].date BETWEEN "2013-01-01 00:00:00 +0100" AND "2014-01-01 00:00:00 +0100";The + same + as:SELECT name, reviews[0].date +FROM `travel-sample` +WHERE type = "hotel" +AND reviews[0].date BETWEEN "2013-01-01 %" AND "2014-01-01 %";Results:[ + { + "date": "2013-06-13 01:39:18 +0300", + "name": "Le Clos Fleuri" + }, + { + "date": "2013-07-02 14:32:55 +0300", + "name": "The George Hotel" + }, + { + "date": "2013-06-22 18:33:50 +0300", + "name": "Medway Youth Hostel" + }, +...When + querying between two dates, you must specify the full date (with time + and time zone) or use the wildcard character (%).     @@ -1095,17 +1083,13 @@
    Examples
    SELECT DATE_FORMAT_STR('2016-05-15T00:00:23+00:00', '1111-11-11') as full_to_short, DATE_FORMAT_STR('2016-05-15', '1111-11-11T00:00:00+00:00') as short_to_full, - DATE_FORMAT_STR('01:10:05', '1111-11-11T01:01:01Z') as time_to_full; - -{ - "results": [ - { - "full_to_short": "2016-05-15", - "short_to_full": "2016-05-15T00:00:00-07:00", - "time_to_full": "0000-01-01T01:10:05-08:00" - } - ] -}
    + DATE_FORMAT_STR('01:10:05', '1111-11-11T01:01:01Z') as time_to_full;Results:[ + { + "full_to_short": "2016-05-15", + "short_to_full": "2016-05-15T00:00:00-07:00", + "time_to_full": "0000-01-01T01:10:05-08:00" + } +] @@ -1161,20 +1145,16 @@ DATE_PART_MILLIS(1463284740000, 'day', 'UTC') as day_utc, DATE_PART_MILLIS(1463284740000, 'month') as month, DATE_PART_MILLIS(1463284740000, 'week') as week, - DATE_PART_MILLIS(1463284740000, 'year') as year; - -{ - "results": [ - { - "day_local": 14, - "day_pst": 14, - "day_utc": 15, - "month": 5, - "week": 20, - "year": 2016 - } - ] -} + DATE_PART_MILLIS(1463284740000, 'year') as year;Results:[ + { + "day_local": 14, + "day_pst": 14, + "day_utc": 15, + "month": 5, + "week": 20, + "year": 2016 + } +] @@ -1217,25 +1197,19 @@
    Examples
    -
    - SELECT DATE_PART_STR('2016-05-15T03:59:00Z', 'day') as day, +
    SELECT DATE_PART_STR('2016-05-15T03:59:00Z', 'day') as day, DATE_PART_STR('2016-05-15T03:59:00Z', 'millisecond') as millisecond, DATE_PART_STR('2016-05-15T03:59:00Z', 'month') as month, DATE_PART_STR('2016-05-15T03:59:00Z', 'week') as week, - DATE_PART_STR('2016-05-15T03:59:00Z', 'year') as year; - -{ - "results": [ - { - "day": 15, - "millisecond": 0, - "month": 5, - "week": 20, - "year": 2016 - } - ] -} -
    + DATE_PART_STR('2016-05-15T03:59:00Z', 'year') as year;Results:[ + { + "day": 15, + "millisecond": 0, + "month": 5, + "week": 20, + "year": 2016 + } +]     @@ -1312,36 +1286,34 @@
    Examples
    -
    - SELECT DATE_RANGE_MILLIS(1480752000000, 1475478000000, 'month', -1); -{ - "results": [ - { - "$1": [ - 1480752000000, - 1478156400000, - 1475478000000 - ] - } +
    Example 1: Range of milliseconds by + month.SELECT DATE_RANGE_MILLIS(1480752000000, 1475478000000, 'month', -1) as Milliseconds;Results:[ + { + "Milliseconds": [ + 1480752000000, + 1478156400000 ] -} - -SELECT ARRAY_MILLIS_TO_STR(a,'2016-12-03') -FOR a -IN DATE_RANGE_MILLIS(1480752000000, 1449129600000, 'month', -1); - -{ - "results": [ - { - "$1": [ - "2016-12-03", - "2016-11-03", - "2016-10-03" - ] - } + } +]Example + 1b: Range of milliseconds by previous + month.SELECT DATE_RANGE_MILLIS(1480752000000, 1449129600000, 'month', -1) as Months;Results:[ + { + "Months": [ + 1480752000000, + 1478156400000, + 1475478000000, + 1472886000000, + 1470207600000, + 1467529200000, + 1464937200000, + 1462258800000, + 1459666800000, + 1456992000000, + 1454486400000, + 1451808000000 ] -} -
    + } +]     @@ -1439,87 +1411,70 @@ IN DATE_RANGE_MILLIS(1480752000000, 1449129600000, 'month', -1);
    Examples
    -
    - SELECT DATE_RANGE_STR('2015-11-30T15:04:05.999', '2017-04-14T15:04:06.998', 'quarter'); - -[ - { - "$1": [ - "2015-11-30T15:04:05.999", - "2016-03-01T15:04:05.999", - "2016-06-01T15:04:05.999", - "2016-09-01T15:04:05.999", - "2016-12-01T15:04:05.999", - "2017-03-01T15:04:05.999" - ] - } -] - -SELECT DATE_RANGE_STR('1111-11-11T15:04:05.999', '2016-01-05T15:04:05.998', 'day', 1); - -[ - { - "$1": [ - "1111-11-11T15:04:05.999", - "2016-01-02T15:04:05.999", - "2016-01-03T15:04:05.999", - "2016-01-04T15:04:05.999" - ] - } -] - -SELECT DATE_RANGE_STR('2018-01-01','2019-01-01', 'month', 4); - -[ - { - "$1": [ - "2018-01-01", - "2018-05-01", - "2018-09-01" - ] - } -] - - -SELECT DATE_RANGE_STR('2016-01-05T15:04:05.999', '1111-11-11T15:04:06.998', 'day', -1)"; - -{ - "results": [ - { - "$1": [ - "2016-01-05T15:04:05.999", - "2016-01-04T15:04:05.999", - "2016-01-03T15:04:05.999", - "2016-01-02T15:04:05.999" - ] - } +
    Example 1: Ranges by + quarters.SELECT DATE_RANGE_STR('2015-11-30T15:04:05.999', '2017-04-14T15:04:06.998', 'quarter') AS Quarters;Results:[ + { + "Quarters": [ + "2015-11-30T15:04:05.999", + "2016-03-01T15:04:05.999", + "2016-06-01T15:04:05.999", + "2016-09-01T15:04:05.999", + "2016-12-01T15:04:05.999", + "2017-03-01T15:04:05.999" ] -} - -SELECT DATE_RANGE_STR('2015-01-01T01:01:01', - DATE_FORMAT_STR('1111-11-11', '1111-11-11T00:00:00'), 'month'); - -{ - "results": [ - { - "$1": [ - "2015-01-01T01:01:01", - "2015-02-01T01:01:01", - "2015-03-01T01:01:01", - "2015-04-01T01:01:01", - "2015-05-01T01:01:01", - "2015-06-01T01:01:01", - "2015-07-01T01:01:01", - "2015-08-01T01:01:01", - "2015-09-01T01:01:01", - "2015-10-01T01:01:01", - "2015-11-01T01:01:01", - "2015-12-01T01:01:01" - ] - } + } +]Example + 2: Ranges by a single + day.SELECT DATE_RANGE_STR('2016-01-01T15:04:05.999', '2016-01-05T15:04:05.998', 'day', 1) as Days;Results:[ + { + "Days": [ + "2016-01-01T15:04:05.999", + "2016-01-02T15:04:05.999", + "2016-01-03T15:04:05.999", + "2016-01-04T15:04:05.999" ] -} -
    + } +]Example + 3: Ranges by four + months.SELECT DATE_RANGE_STR('2018-01-01','2019-01-01', 'month', 4) as Months;Results:[ + { + "Months": [ + "2018-01-01", + "2018-05-01", + "2018-09-01" + ] + } +]Example + 4: Ranges by previous + days.SELECT DATE_RANGE_STR('2016-01-05T15:04:05.999', '2016-01-01T15:04:06.998', 'day', -1) as Previous;Results:[ + { + "Previous": [ + "2016-01-05T15:04:05.999", + "2016-01-04T15:04:05.999", + "2016-01-03T15:04:05.999", + "2016-01-02T15:04:05.999" + ] + } +]Example + 5: Ranges by + month.SELECT DATE_RANGE_STR('2015-01-01T01:01:01', '2015-12-11T00:00:00', 'month', 1) as Months;Results:[ + { + "Months": [ + "2015-01-01T01:01:01", + "2015-02-01T01:01:01", + "2015-03-01T01:01:01", + "2015-04-01T01:01:01", + "2015-05-01T01:01:01", + "2015-06-01T01:01:01", + "2015-07-01T01:01:01", + "2015-08-01T01:01:01", + "2015-09-01T01:01:01", + "2015-10-01T01:01:01", + "2015-11-01T01:01:01", + "2015-12-01T01:01:01" + ] + } +]     @@ -1564,21 +1519,15 @@ SELECT DATE_RANGE_STR('2015-01-01T01:01:01',
    Examples
    -
    - SELECT DATE_TRUNC_MILLIS(1463284740000, 'day') as day, +
    SELECT DATE_TRUNC_MILLIS(1463284740000, 'day') as day, DATE_TRUNC_MILLIS(1463284740000, 'month') as month, - DATE_TRUNC_MILLIS(1463284740000, 'year') as year; - -{ - "results": [ - { - "day": 1463270400000, - "month": 1462147200000, - "year": 1451696400000 - } - ] -} -
    + DATE_TRUNC_MILLIS(1463284740000, 'year') as year;Results:[ + { + "day": 1463270400000, + "month": 1462147200000, + "year": 1451696400000 + } +]     @@ -1620,21 +1569,15 @@ SELECT DATE_RANGE_STR('2015-01-01T01:01:01',
    Examples
    -
    - SELECT DATE_TRUNC_STR('2016-05-18T03:59:00Z', 'day') as day, +
    SELECT DATE_TRUNC_STR('2016-05-18T03:59:00Z', 'day') as day, DATE_TRUNC_STR('2016-05-18T03:59:00Z', 'month') as month, - DATE_TRUNC_STR('2016-05-18T03:59:00Z', 'year') as year; - -{ - "results": [ - { - "day": "2016-05-18T00:00:00Z", - "month": "2016-05-01T00:00:00Z", - "year": "1111-11-11T00:00:00Z" - } - ] -} -
    + DATE_TRUNC_STR('2016-05-18T03:59:00Z', 'year') as year;Results:[ + { + "day": "2016-05-18T00:00:00Z", + "month": "2016-05-01T00:00:00Z", + "year": "2016-01-01T00:00:00Z" + } +]     @@ -1667,21 +1610,15 @@ SELECT DATE_RANGE_STR('2015-01-01T01:01:01',
    Examples
    -
    - SELECT DURATION_TO_STR(2000) as microsecs, +
    SELECT DURATION_TO_STR(2000) as microsecs, DURATION_TO_STR(2000000) as millisecs, - DURATION_TO_STR(2000000000) as secs; - -{ - "results": [ - { - "microsecs": "2µs", - "millisecs": "2ms", - "secs": "2s" - } - ] -} -
    + DURATION_TO_STR(2000000000) as secs;Results:[ + { + "microsecs": "2µs", + "millisecs": "2ms", + "secs": "2s" + } +]     @@ -1715,16 +1652,11 @@ SELECT DATE_RANGE_STR('2015-01-01T01:01:01',
    Examples
    -
    - SELECT MILLIS("2016-05-15T03:59:00Z"); -{ - "results": [ - { - "$1": 1463284740000 - } - ] -} -
    +
    SELECT MILLIS("2016-05-15T03:59:00Z") as DateStringInMilliseconds;Results:[ + { + "DateStringInMilliseconds": 1463284740000 + } +]
     @@ -1777,21 +1709,15 @@ SELECT DATE_RANGE_STR('2015-01-01T01:01:01',
    Examples
    -
    - SELECT MILLIS_TO_STR(1463284740000) as full_date, +
    SELECT MILLIS_TO_STR(1463284740000) as full_date, MILLIS_TO_STR(1463284740000, 'invalid format') as invalid_format, - MILLIS_TO_STR(1463284740000, '1111-11-11') as short_date; - -{ - "results": [ - { - "full_date": "2016-05-14T20:59:00-07:00", - "invalid_format": "2016-05-14T20:59:00-07:00", - "short_date": "2016-05-14" - } - ] -} -
    + MILLIS_TO_STR(1463284740000, '1111-11-11') as short_date;Results:[ + { + "full_date": "2016-05-14T20:59:00-07:00", + "invalid_format": "2016-05-14T20:59:00-07:00", + "short_date": "2016-05-14" + } +]     @@ -1846,22 +1772,15 @@ SELECT DATE_RANGE_STR('2015-01-01T01:01:01',
    Examples
    -
    - SELECT MILLIS_TO_TZ(1463284740000, 'America/New_York') as est, +
    SELECT MILLIS_TO_TZ(1463284740000, 'America/New_York') as est, MILLIS_TO_TZ(1463284740000, 'Asia/Kolkata') as ist, - MILLIS_TO_TZ(1463284740000, 'UTC') as utc; - -{ - "results": [ - { - "est": "2016-05-14T23:59:00-04:00", - "invalid_tz": "2016-05-15T03:59:00Z", - "ist": "2016-05-15T09:29:00+05:30", - "utc": "2016-05-15T03:59:00Z" - } - ] -} -
    + MILLIS_TO_TZ(1463284740000, 'UTC') as utc;Results:[ + { + "est": "2016-05-14T23:59:00-04:00", + "ist": "2016-05-15T09:29:00+05:30", + "utc": "2016-05-15T03:59:00Z" + } +]     @@ -1905,21 +1824,15 @@ SELECT DATE_RANGE_STR('2015-01-01T01:01:01',
    Examples
    -
    - SELECT MILLIS_TO_UTC(1463284740000) as full_date, +
    SELECT MILLIS_TO_UTC(1463284740000) as full_date, MILLIS_TO_UTC(1463284740000, 'invalid format') as invalid_format, - MILLIS_TO_UTC(1463284740000, '1111-11-11') as short_date; - -{ - "results": [ - { - "full_date": "2016-05-15T03:59:00Z", - "invalid_format": "2016-05-15T03:59:00Z", - "short_date": "2016-05-15" - } - ] -} -
    + MILLIS_TO_UTC(1463284740000, '1111-11-11') as short_date;Results:[ + { + "full_date": "2016-05-15T03:59:00Z", + "invalid_format": "2016-05-15T03:59:00Z", + "short_date": "2016-05-15" + } +]     @@ -1961,38 +1874,27 @@ SELECT DATE_RANGE_STR('2015-01-01T01:01:01',
    Examples
    -
    - SELECT NOW_LOCAL() as full_date, +
    Example 1: Various arguments of + NOW_LOCAL().SELECT NOW_LOCAL() as full_date, NOW_LOCAL('invalid date') as invalid_date, - NOW_LOCAL('1111-11-11') as short_date; - -{ - "results": [ - { - "full_date": "2017-02-03T03:10:19.01-08:00", - "invalid_date": "2017-02-03T03:10:19.01-08:00", - "short_date": "2017-02-03" - } - ] -} - -SELECT NOW_LOCAL(), NOW_LOCAL(), NOW_LOCAL(), NOW_LOCAL(), - NOW_LOCAL(), NOW_LOCAL(), CLOCK_LOCAL(); - -{ - "results": [ - { - "$1": "2017-02-02T06:56:09.67-08:00", - "$2": "2017-02-02T06:56:09.67-08:00", - "$3": "2017-02-02T06:56:09.67-08:00", - "$4": "2017-02-02T06:56:09.67-08:00", - "$5": "2017-02-02T06:56:09.67-08:00", - "$6": "2017-02-02T06:56:09.67-08:00", - "$7": "2017-02-02T06:56:09.673-08:00" - } - ] -} -
    + NOW_LOCAL('1111-11-11') as short_date;Results:[ + { + "full_date": "2018-01-23T14:03:40.26-08:00", + "invalid_date": "2018-01-23T14:03:40.26-08:00", + "short_date": "2018-01-23" + } +]Example + 2: Difference between NOW_LOCAL() and + CLOCK_LOCAL().SELECT NOW_LOCAL(), NOW_LOCAL(), NOW_LOCAL(), NOW_LOCAL(), NOW_LOCAL(), CLOCK_LOCAL();Results:[ + { + "$1": "2018-01-23T14:06:20.254-08:00", + "$2": "2018-01-23T14:06:20.254-08:00", + "$3": "2018-01-23T14:06:20.254-08:00", + "$4": "2018-01-23T14:06:20.254-08:00", + "$5": "2018-01-23T14:06:20.254-08:00", + "$6": "2018-01-23T14:06:20.256-08:00" + } +]     @@ -2020,33 +1922,21 @@ SELECT NOW_LOCAL(), NOW_LOCAL(), NOW_LOCAL(), NOW_LOCAL(),
    Examples
    -
    - SELECT NOW_MILLIS(); -{ - "results": [ - { - "$1": 1486046939117.3264 - } - ] -} - -SELECT NOW_MILLIS(), NOW_MILLIS(), NOW_MILLIS(), NOW_MILLIS(), - NOW_MILLIS(), NOW_MILLIS(), CLOCK_MILLIS(); - -{ - "results": [ - { - "$1": 1486046983987.5046, - "$2": 1486046983987.5046, - "$3": 1486046983987.5046, - "$4": 1486046983987.5046, - "$5": 1486046983987.5046, - "$6": 1486046983987.5046, - "$7": 1486046983987.6514 - } - ] -} -
    +
    Example 1: The time now in + milliseconds.SELECT NOW_MILLIS() as NowInMilliseconds;Results:[ + { + "NowInMilliseconds": 1516745378065.12 + } +]Example + 2: Difference between NOW_MILLIS() and + CLOCK_MILLIS().SELECT NOW_MILLIS(), NOW_MILLIS(), NOW_MILLIS(), NOW_MILLIS(), CLOCK_MILLIS();Results:[ + { + "$1": 1516745528579.607, + "$2": 1516745528579.607, + "$3": 1516745528579.607, + "$4": 1516745528580.29 + } +]
     @@ -2098,41 +1988,27 @@ SELECT NOW_MILLIS(), NOW_MILLIS(), NOW_MILLIS(), NOW_MILLIS(),
    Examples
    -
    - SELECT NOW_TZ('invalid tz') as invalid_tz, +
    Example 1: Various arguments for + NOW_TZ().SELECT NOW_TZ('invalid tz') as invalid_tz, NOW_TZ('Asia/Kolkata') as ist, NOW_TZ('UTC') as utc, - NOW_TZ('UTC', '1111-11-11') as utc_short_date; - -{ - "results": [ - { - "invalid_tz": null, - "ist": "2017-02-03T16:42:42.686+05:30", - "utc": "2017-02-03T11:12:42.686Z", - "utc_short_date": "2017-02-03" - } - ] -} - -SELECT NOW_TZ('UTC'), NOW_TZ('UTC'), NOW_TZ('UTC'), NOW_TZ('UTC'), - NOW_TZ('UTC'), NOW_TZ('UTC'), NOW_TZ('UTC'), CLOCK_TZ('UTC'); - -{ - "results": [ - { - "$1": "2017-02-03T11:15:44.457Z", - "$2": "2017-02-03T11:15:44.457Z", - "$3": "2017-02-03T11:15:44.457Z", - "$4": "2017-02-03T11:15:44.457Z", - "$5": "2017-02-03T11:15:44.457Z", - "$6": "2017-02-03T11:15:44.457Z", - "$7": "2017-02-03T11:15:44.457Z", - "$8": "2017-02-03T11:15:44.459Z" - } - ] -} -
    + NOW_TZ('UTC', '1111-11-11') as utc_short_date;Results:[ + { + "invalid_tz": null, + "ist": "2018-01-24T03:43:36.457+05:30", + "utc": "2018-01-23T22:13:36.457Z", + "utc_short_date": "2018-01-23" + } +]Example + 2: Difference between NOW_TZ() and + CLOCK_TZ().SELECT NOW_TZ('UTC'), NOW_TZ('UTC'), NOW_TZ('UTC'), CLOCK_TZ('UTC');Results:[ + { + "$1": "2018-01-23T22:15:59.551Z", + "$2": "2018-01-23T22:15:59.551Z", + "$3": "2018-01-23T22:15:59.551Z", + "$4": "2018-01-23T22:15:59.552Z" + } +]     @@ -2174,37 +2050,28 @@ SELECT NOW_TZ('UTC'), NOW_TZ('UTC'), NOW_TZ('UTC'), NOW_TZ('UTC'),
    Examples
    -
    - SELECT NOW_STR() as full_date, +
    Example 1: Various arguments for + NOW_STR().SELECT NOW_STR() as full_date, NOW_STR('invalid date') as invalid_date, - NOW_STR('1111-11-11') as short_date; - -{ - "results": [ - { - "full_date": "2017-02-03T03:21:29.814-08:00", - "invalid_date": "2017-02-03T03:21:29.814-08:00", - "short_date": "2017-02-03" - } - ] -} - -SELECT NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), CLOCK_STR(); - -{ - "results": [ - { - "$1": "2017-02-03T03:22:34.486-08:00", - "$2": "2017-02-03T03:22:34.486-08:00", - "$3": "2017-02-03T03:22:34.486-08:00", - "$4": "2017-02-03T03:22:34.486-08:00", - "$5": "2017-02-03T03:22:34.486-08:00", - "$6": "2017-02-03T03:22:34.486-08:00", - "$7": "2017-02-03T03:22:34.488-08:00" - } - ] -} -
    + NOW_STR('1111-11-11') as short_date;Results:[ + { + "full_date": "2018-01-23T14:16:58.075-08:00", + "invalid_date": "2018-01-23T14:16:58.075-08:00", + "short_date": "2018-01-23" + } +]Example + 2: Difference between NOW_STR() and + CLOCK_STR().SELECT NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), CLOCK_STR();Results:[ + { + "$1": "2018-01-23T14:18:37.605-08:00", + "$2": "2018-01-23T14:18:37.605-08:00", + "$3": "2018-01-23T14:18:37.605-08:00", + "$4": "2018-01-23T14:18:37.605-08:00", + "$5": "2018-01-23T14:18:37.605-08:00", + "$6": "2018-01-23T14:18:37.605-08:00", + "$7": "2018-01-23T14:18:37.607-08:00" + } +]     @@ -2247,34 +2114,25 @@ SELECT NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), NOW_STR(), CLOCK_S
    Examples
    -
    - SELECT NOW_UTC(); - -{ - "results": [ - { - "$1": "2017-02-03T11:16:54.419Z" - } - ] -} - -SELECT NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), CLOCK_UTC(); - -{ - "results": [ - { - "$1": "2017-02-03T11:18:27.183Z", - "$2": "2017-02-03T11:18:27.183Z", - "$3": "2017-02-03T11:18:27.183Z", - "$4": "2017-02-03T11:18:27.183Z", - "$5": "2017-02-03T11:18:27.183Z", - "$6": "2017-02-03T11:18:27.183Z", - "$7": "2017-02-03T11:18:27.183Z", - "$8": "2017-02-03T11:18:27.185Z" - } - ] -} -
    +
    Example 1: The current UTC + time.SELECT NOW_UTC() as CurrentUTC;Results:[ + { + "CurrentUTC": "2018-01-23T22:20:43.971Z" + } +]Example + 2: Difference between NOW_UTC() and + CLOCK_UTC().SELECT NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), CLOCK_UTC();Results:[ + { + "$1": "2018-01-23T22:21:46.769Z", + "$2": "2018-01-23T22:21:46.769Z", + "$3": "2018-01-23T22:21:46.769Z", + "$4": "2018-01-23T22:21:46.769Z", + "$5": "2018-01-23T22:21:46.769Z", + "$6": "2018-01-23T22:21:46.769Z", + "$7": "2018-01-23T22:21:46.769Z", + "$8": "2018-01-23T22:21:46.77Z" + } +]
     @@ -2315,27 +2173,21 @@ SELECT NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC(), NOW_UTC
    Examples
    -
    - SELECT STR_TO_DURATION('1h') as hour, +
    SELECT STR_TO_DURATION('1h') as hour, STR_TO_DURATION('1us') as microsecond, STR_TO_DURATION('1ms') as millisecond, STR_TO_DURATION('1m') as minute, STR_TO_DURATION('1ns') as nanosecond, -STR_TO_DURATION('1s') as second; - -{ - "results": [ - { - "hour": 3600000000000, - "microsecond": 1000, - "millisecond": 1000000, - "minute": 60000000000, - "nanosecond": 1, - "second": 1000000000 - } - ] -} -
    +STR_TO_DURATION('1s') as second;Results:[ + { + "hour": 3600000000000, + "microsecond": 1000, + "millisecond": 1000000, + "minute": 60000000000, + "nanosecond": 1, + "second": 1000000000 + } +]     @@ -2371,17 +2223,11 @@ STR_TO_DURATION('1s') as second;
    Examples
    -
    - SELECT STR_TO_MILLIS("2016-05-15T03:59:00Z"); - -{ - "results": [ - { - "$1": 1463284740000 - } - ] -} -
    +
    SELECT STR_TO_MILLIS("2016-05-15T03:59:00Z") as Milliseconds;Results:[ + { + "Milliseconds": 1463284740000 + } +]
     @@ -2416,19 +2262,13 @@ STR_TO_DURATION('1s') as second;
    Examples
    -
    - SELECT STR_TO_UTC('1111-11-11T00:00:00+08:00') as full_date, -STR_TO_UTC('1111-11-11') as short_date; - -{ - "results": [ - { - "full_date": "2015-12-31T16:00:00Z", - "short_date": "1111-11-11" - } - ] -} -
    +
    SELECT STR_TO_UTC('1111-11-11T00:00:00+08:00') as full_date, +STR_TO_UTC('1111-11-11') as short_date;Results:[ + { + "full_date": "1111-11-10T16:00:00Z", + "short_date": "1111-11-11" + } +]
     @@ -2473,21 +2313,15 @@ STR_TO_UTC('1111-11-11') as short_date;
    Examples
    -
    - SELECT STR_TO_TZ('1111-11-11T00:00:00+08:00', 'America/New_York') as est, +
    SELECT STR_TO_TZ('1111-11-11T00:00:00+08:00', 'America/New_York') as est, STR_TO_TZ('1111-11-11T00:00:00+08:00', 'UTC') as utc, - STR_TO_TZ('1111-11-11', 'UTC') as utc_short; - -{ - "results": [ - { - "est": "2015-12-31T11:00:00-05:00", - "utc": "2015-12-31T16:00:00Z", - "utc_short": "1111-11-11" - } - ] -} -
    + STR_TO_TZ('1111-11-11', 'UTC') as utc_short;Results:[ + { + "est": "1111-11-10T11:00:00-05:00", + "utc": "1111-11-10T16:00:00Z", + "utc_short": "1111-11-11" + } +]     @@ -2533,17 +2367,11 @@ STR_TO_UTC('1111-11-11') as short_date;
    Examples
    -
    - SELECT WEEKDAY_MILLIS(1486237655742, 'America/Tijuana'); - -{ - "results": [ - { - "$1": "Saturday" - } -] - -
    +
    SELECT WEEKDAY_MILLIS(1486237655742, 'America/Tijuana') as Day;Results:[ + { + "Day": "Saturday" + } +]
     @@ -2581,16 +2409,11 @@ STR_TO_UTC('1111-11-11') as short_date;
    Examples
    -
    - SELECT WEEKDAY_STR('2017-02-05'); - -[ - { - "$1": "Sunday" - } -] - -
    +
    SELECT WEEKDAY_STR('2017-02-05') as Day;Results:[ + { + "Day": "Sunday" + } +]
     diff --git a/content/n1ql/n1ql-language-reference/from.dita b/content/n1ql/n1ql-language-reference/from.dita index 6d47fea7f1..17bbe97dc0 100644 --- a/content/n1ql/n1ql-language-reference/from.dita +++ b/content/n1ql/n1ql-language-reference/from.dita @@ -1,249 +1,309 @@ - - + + FROM clause + The FROM clause specifies the keyspaces and JOIN operations on + them. - -

    The FROM clause of a SELECT query or subquery defines the - keyspaces and the source of input documents or objects for the query. Every - FROM clause specifies one or more keyspaces. The first keyspace is - called the primary keyspace. This is an optional clause for your query. If this clause - is omitted, the input for the query is a single empty object. You can perform - calculations with the SELECT statement if the FROM - clause is omitted.

    -

    -

    The from-term defines the input objects for the query and it can be - either a keyspace identifier or an expression. When using nested subqueries, the - from-term of the outermost parent query can only have a keyspace - identifier, a subquery, or a constant - expression since the from-term needs to independently - produce input documents for the query. However, the subqueries can use generic variable - expressions, which may be dependent on the . For more details, see .

    -

    - Couchbase Server version 4.x supports only keyspace identifier or a subquery in - the from-term, but not expressions. Couchbase Server version 4.6.2 - adds support for generic expression in the from-term. -

    -

    - -

    -

    - -FROM from-term - -

    where from-term has the following syntax:

    - - from-term::= keyspace-ref [ [ AS ] alias ] [ use-keys-clause ] - | "(" select ")" [ AS ] alias - | expr [ AS ] alias - | from-term ( join-clause | nest-clause | unnest-clause ) - -

    from-path:

    - [ namespace : ] path - -

    namespace:

    - identifier -

    expr:

    - see N1QL expression - -

    use-keys-clause:

    - USE [ PRIMARY ] KEYS expression - -

    join-clause:

    - ( lookup-join | index-join ) - -

    lookup-join:

    - [ join-type ] JOIN from-path [ [ AS ] alias ] on-keys-clause - -

    index-join:

    - [ join-type ] JOIN from-path [ [ AS ] alias ] ( on-keys-clause | on-key-for-clause ) - -

    join-type:

    - INNER | LEFT [ OUTER ] - -

    on-keys-clause:

    - ON [ PRIMARY ] KEYS expression - -

    on-key-for-clause:

    - ON [ PRIMARY ] KEY rhs-expression.lhs-expression-key FOR lhs-expression -

    rhs-expression: keyspace or expression corresponding to the right hand side of JOIN.

    -

    lhs-expression: keyspace or expression corresponding to the left hand side of JOIN.

    -

    lhs-expression-key: attribute in rhs-expression - referencing primary key for lhs-expression.

    - -

    nest-clause:

    - [ join-type ] NEST from-path [ ( [ AS ] alias ) ] on-keys-clause - -

    unnest-clause:

    - [ join-type ] [ UNNEST | FLATTEN ] expression [ ( [ AS ] alias ) ] - -

    Omitted FROM clause

    -

    If the FROM clause is omitted, the data source is equivalent to an array - containing a single empty object. This allows you to evaluate expressions that do not - depend on stored data.

    -

    Evaluating an expression SELECT 10 + 20 produces the following result:

    -[ { "$1" : 30 } ] -

    Counting the number of inputs SELECT COUNT(*) AS input_count produces the - following result:

    -[ { "input_count" : 1 } ] -

    Getting the input contents SELECT * produces the following result:

    - [ { } ] - -

    Keyspaces

    -

    The simplest type of FROM clause specifies a keyspace:

    - SELECT * FROM customer -

    This returns every value in the customer keyspace.

    -

    The keyspace can be prefixed with an optional namespace (pool):

    - SELECT * FROM main:customer -

    This queries the customer keyspace in the main namespace.

    -

    If the namespace is omitted, the default namespace in the current session is used.

    -

    -

    Keyspace Identifier

    This is the name or - identifier of an independent keyspace that can serve as a data source or keyspace of - one or more documents. Such keyspaces are not dependent on any of the .

    The following example Q1 - uses the keyspace `travel-sample`.

    - Example Q1: - -SELECT t1.city -FROM `travel-sample` t1 -WHERE t1.type = "landmark" LIMIT 4; -

    The following non-correlated subquery example Q2 uses the - keyspace `travel-sample` independent of the same keyspace used - in the outer query:

    - Example Q2: Find cities that have landmarks and airports. - -SELECT t1.city -FROM `travel-sample` t1 -WHERE t1.type = "landmark" AND - t1.city IN (SELECT RAW city - FROM `travel-sample` - WHERE type = "airport"); -

    The following correlated subquery Q2A uses keyspace alias from outer - query.

    - Example Q2A: Find cities that have museum landmarks. - -SELECT t1.city, t1.name -FROM `travel-sample` t1 -WHERE t1.type = "landmark" AND - (SELECT raw t2 - FROM split(t1.name) t2 - WHERE t2 = "museum")[0] is not null; -

    The following subquery example Q2B uses different keyspaces in - outer/inner queries.

    - Example Q2B: Find cities that have landmarks and breweries. - -SELECT t1.city -FROM `travel-sample` t1 -WHERE t1.type = "landmark" AND - t1.city IN (SELECT RAW b1.city - FROM `beer-sample` b1 - WHERE b1.type = "brewery"); -

    -
    - N1QL Expression -

    Couchbase Server version 4.6.2 adds support for generic expressions in the - from-term. This is a very powerful functionality as it enables usage of - various N1QL functions, operators, path expressions, language constructs on constant - expressions, variables, and subqueries.

      + +

      (Introduced in Couchbase Server 4.0)

      +
      + Purpose +

      In a SELECT query or subquery, the FROM clause + specifies one or more of the following:

        +
      • Keyspaces
        • +
      • Subqueries (such as derived tables)
        • +
      • JOIN clauses
        • +
      • JOIN conditions
        • +
      • Expressions (nested collections, CURL(), or other + expressions)
        • +

      +
      +
      + Prerequisites +

      For you to select data from keyspace or expression, you must have the + query_select privilege on that keyspace. For more details + about user roles, see .

      +
      +
      Syntax

      + +

      + FROM from-keyspace [ [ AS ] alias1 ] [ USE KEYS use-clause ] + | "(" SELECT ")" [ [ AS ] alias2 ] + | expr [ [ AS ] alias3 ] + | from-term ( join-clause | nest-clause | unnest-clause ) +

      Arguments + + + + + + Argument / Clause + Description + + + + + from-keyspace

      +

      + +
      +
      + +
      +

      . . . AS alias

      +

      + +
      +
      + +
      +

      . . . USE KEYS + use-clause

       + Identifier that + represents the keyspace for the query, such as FROM + `travel-sample`

      +

      + +
      +
      + +
      +

      To assign a name to a keyspace or expression.

      +

      + +
      +
      + +
      +

      To specify one or more document keys.

       +       + + ( + select-expr ) + To specify a N1QL SELECT subquery + + + expr + A N1QL expression generating JSON documents or objects. + + + from-term
      + +
      +
      + +

      +

      + +
      +
      + +
      +

      . . . JOIN join-clause

      +

      + +
      +
      + +
      +
      + +
      +
      +
      + +
      +
      + +
      +
      +       +
      +

      . . . NEST nest-clause

      +

      + +
      +
      + +
      +
      + +
      +
      +
      + +
      +
      + +
      +
      +       +
      +

      . . . UNNEST + unnest-clause

       + A N1QL expression that defines the input object(s) for the query, + which can be either a keyspace identifier, generic expression, or + subquery along with + one or more JOIN, NEST, or UNNEST clause.

      +

      + +
      +
      + +
      +

      Create an input object by combining two or more source + objects via ANSI JOIN, Lookup JOIN, or Index JOIN.

      +

      + +
      +
      + +
      +

      Create an input object by producing a single result of + nesting keyspaces via ANSI NEST, Lookup NEST, or Index NEST.

      +

      + +
      +
      + +
      +

      Create an input object by flattening an array in the parent + document.

       +       + + +
      +
      +
      + from-keyspace +

      You can specify a keyspace to query from, either a specific bucket or a constant + expression. If the from-keyspace clause is used, then there must + be a from-keyspace-name specified.

      +

      The simplest type of from-keyspace clause specifies a single bucket (i.e., + SELECT * FROM `travel-sample`).

      +

      Keyspace identifier is the name or identifier of an independent place for a data + source of one or more documents. Such keyspaces are not dependent on any of the + Variable in Scope of a + Subquery.

      +

      +

      from-keyspace Example: Use a keyspace from a single bucket.

      +

      Select four unique landmarks from the `travel-sample` bucket.

      +

      SELECT DISTINCT name +FROM `travel-sample` +WHERE type = "landmark" +LIMIT 4;Results:

      + [ + { + "name": "Royal Engineers Museum" + }, + { + "name": "Hollywood Bowl" + }, + { + "name": "Thai Won Mien" + }, + { + "name": "Spice Court" + } +] +

      N1QL Expressions in a FROM Clause

      +

      Couchbase Server version 4.6.2 added support for generic expressions in the + from-term clause; and this adds huge flexibility by the + enabling of various N1QL functions, operators, path expressions, language constructs + on constant expressions, variables, and subqueries to create just about any FROM + clause imaginable.

      • -

        When the from-term is an expression, USE KEYS - or USE INDEX clauses are not allowed.

        +

        When the from-term is an expression, + USE KEYS or USE INDEX clauses are + not allowed.

      • -

        When using JOIN clause, NEST clause, or UNNEST clause, the left - side keyspace can be an expression or subquery, but the right side +

        When using a JOIN clause, + NEST clause, or UNNEST clause, the + left-side keyspace can be an expression or subquery, but the right-side keyspace must be a keyspace identifier.

        -
        Independent Constant Expression
        +
        1. Independent Constant Expression
        This includes any N1QL expressions of JSON scalar values, static - JSON literals, objects, or N1QL functions. For example:

        - SELECT * FROM [1, 2, "name", { "type" : "airport", "id" : "SFO"}] AS ks1 -SELECT * FROM CURL(...) as ks2; + JSON literals, objects, or N1QL functions, for example:

        + SELECT * FROM [1, 2, "name", { "type" : "airport", "id" : "SFO"}] AS ks1; + +SELECT CURL("https://maps.googleapis.com/maps/api/geocode/json", + {"data":"address=Half+Moon+Bay" , "request":"GET"} );

        Note that functions such as CURL() can independently produce input data objects for the query. - However, other N1QL functions can also be used in the + Similarly, other N1QL functions can also be used in the expressions.

         -
        Variable N1QL Expression
        +
        2. Variable N1QL Expression
        -
        This includes expressions that refers to any This includes expressions that refer to any variables - in scope for the query. This is applicable to only - subqueries because the outermost level query cannot use any - variables in its own FROM clause. This makes the subquery + in scope for the query, for + example:SELECT count(*) +FROM `travel-sample` t +LET x = t.geo +WHERE (SELECT RAW y.alt FROM x y)[0] > 6000;The + FROM x clause is an expression that refers to the + outer query. This is applicable to only subqueries because the + outermost level query cannot use any variables in its own + FROM clause. This makes the subquery correlated with outer queries, as explained in the section.
        -
        Subquery and Subquery Expressions
        +
        3. Subquery and Subquery Expressions
        -
        A subquery itself can appear as from-term - expression. In this case, the subquery results are fed as data - source to the outer query. Further, subqueries can occur as a - subquery, or as constituent part of a bigger N1QL expression.

        For example, the following example Q3 shows a - simple case of using subquery in FROM clause, though N1QL is - powerful enough to express it without a subquery (as shown - in example Q3A):

        - Example Q3: Find highest altitude airports/cities in each country - -SELECT t1.country, t1.max_country_alt, - ARRAY x.city FOR x IN t1.c - WHEN x.alt = t1.max_country_alt END -FROM (SELECT country, array_agg({"alt": geo.alt , city}) c, - max_country_alt - FROM `travel-sample` - WHERE type = "airport" - GROUP BY country -LETTING max_country_alt = max(geo.alt) ) t1; - Example Q3A: Query Q3 without using subqueries - -SELECT country, - (ARRAY x.city FOR x IN array_agg({"alt": geo.alt, city}) - WHEN x.alt = max_country_alt END) AS max_alt_city, - max_country_alt -FROM `travel-sample` -WHERE type = "airport" -GROUP BY country -LETTING max_country_alt = max(geo.alt); -

        A more realistic example of needing subquery in the - FROM clause arises in more complicated - scenarios such as when multiple levels of aggregates (sort - orders and limits) are required, or when the subquery results of - one keyspace may need to be JOINed with another keyspace.
        -
        For example, the following example Q4 finds for each country, - total number of airports at different altitudes, and the total - number of corresponding cities with those airports. In this - case, the inner query finds first level of grouping of different - altitudes by country, and corresponding number of cities. Then - the outer query builds on the inner query results to count - number of different altitude groups for each country, and the - total number of cities.

        - Example Q4: - -SELECT t1.country, num_alts, total_cities +

        +

        Subquery Example: For each country, find the + number of airports at different altitudes and their + corresponding cities.

        +

        In this case, the inner query finds the first level of + grouping of different altitudes by country and corresponding + number of cities. Then the outer query builds on the inner + query results to count the number of different altitude + groups for each country and the total number of cities.

        +

        SELECT t1.country, num_alts, total_cities FROM (SELECT country, geo.alt AS alt, count(city) AS num_cities FROM `travel-sample` WHERE type = "airport" GROUP BY country, geo.alt) t1 GROUP BY t1.country -LETTING num_alts = count(t1.alt), total_cities = sum(t1.num_cities); - -[ +LETTING num_alts = count(t1.alt), total_cities = sum(t1.num_cities);Results:[ + { + "country": "United States", + "num_alts": 946, + "total_cities": 1560 + }, { "country": "United Kingdom", "num_alts": 128, @@ -253,17 +313,13 @@ LETTING num_alts = count(t1.alt), total_cities = sum(t1.num_cities); "country": "France", "num_alts": 196, "total_cities": 221 - }, - { - "country": "United States", - "num_alts": 946, - "total_cities": 1560 } -] -

        This is equivalent to blending the results of the following - two queries by country, but the subquery in the - from-term simplified it - above.SELECT country,count(city) AS num_cities +]

        +
        +
        This is equivalent to blending the results of the following two + queries by country, but the subquery in the + from-term above simplified + it.SELECT country,count(city) AS num_cities FROM `travel-sample` WHERE type = "airport" GROUP BY country; @@ -272,226 +328,1162 @@ SELECT country, count(distinct geo.alt) AS num_alts FROM `travel-sample` WHERE type = "airport" GROUP BY country; - See for details and - examples.
        + For more details and examples, see and ( + select-expr ).         - -
        Nested Path Expressions
        -
        -
        Expressions used in the from-term can have - nested paths, including constant or variable or subquery - expressions. Similarly, variable expressions are allowed only - subqueries and not in outermost parent queries.
        -
        Further, the nested path variable expression in subquery - from-term must resolve to variables/aliases - in scope, and not to any keyspace identifiers. Otherwise, it - results in a syntax error. Note that, when an expression cannot - be resolved to any variables in scope, it is considered keyspace - identifier. See Nested Paths - in Subqueries for more details.

        The - following example shows usage of nested path over subquery - expression:

        - SELECT x.geo.alt -FROM (SELECT geo from `travel-sample` - WHERE type = "airport") AS x -LIMIT 2; - -[ - { - "alt": 12 - }, +

        +
        • +

      +
      +
      + AS Alias +

      To use a shorter or clearer name anywhere in the query, like SQL, N1QL allows + renaming fields by using the AS keyword to assign an alias to a keyspace or field in + the FROM clause.

      +

      Syntax[AS] alias

      +

      Arguments

      +

      +

      + +
      AS
      +
      [Optional] Reserved word denoting the next word is an alias of the + previous term.
      +       +
      +
      + +
      alias
      +
      [Required if AS is used] String to assign a name to a + keyspace, such as the following equivalent FROM clauses + with and without the AS keyword: + + + + + + FROM `travel-sample` + AS t + FROM `travel-sample` + t + + + FROM `travel-sample` + AS + h

      INNER JOIN + `travel-sample` AS + l

      ON + (h.city = + l.city)

       + FROM `travel-sample` + h

      INNER + JOIN `travel-sample` + l

      ON + (h.city = + l.city)

       +       + + +
      Since the original name may lead to referencing wrong data + and wrong results, you must use the alias name throughout the query + instead of the original keyspace name.

      In the FROM clause, the + renaming appears only in the projection and not the fields + themselves.

      When no alias is used, the keyspace or last + field name of an expression is given as the implicit + alias.

      When an alias conflicts with a keyspace or field + name in the same scope, the identifier always refers to the + alias. This allows for consistent behavior in scenarios where an + identifier only conflicts in some documents. For more + information on aliases, see .

      +       +
      +

      +
      +
      USE KEYS Clause

      You can refer to a document's + unique document key by using the USE KEYS clause. Only documents + having those document keys will be included as inputs to a + query.

      Syntax

      USE [ PRIMARY ] KEYS expr

      + +
      Arguments
      +
      +
      + +
      PRIMARY
      +
      [Optional] USE KEYS and USE PRIMARY + KEYS are synonyms.
      +       +
      +
      + +
      expr
      +
      String of a document key or an array of comma-separated + document keys.
      +       +
      +
      +       +
      USE KEYS Example 1: Select a single document by its + document + key.

      SELECT * +FROM `travel-sample` +USE KEYS "airport_1254";Results:[ { - "alt": 295 + "travel-sample": { + "airportname": "Calais Dunkerque", + "city": "Calais", + "country": "France", + "faa": "CQF", + "geo": { + "alt": 12, + "lat": 50.962097, + "lon": 1.954764 + }, + "icao": "LFAC", + "id": 1254, + "type": "airport", + "tz": "Europe/Paris" + } } -] -

      The following example shows usage of nested path over - constant - expression:SELECT x.b FROM - [{"a" : 1, "b" : {"c" : 2}}, - {"a" : 3, "b" : {"d" : 4}}] AS x -LIMIT 2; - -[ +]

      USE + KEYS Example 2: Select multiple documents by their document + keys.

      SELECT * +FROM `travel-sample` +USE KEYS ["airport_1254","airport_1255"];Results:[ { - "x": { - "c": 2 + "travel-sample": { + "airportname": "Calais Dunkerque", + "city": "Calais", + "country": "France", + "faa": "CQF", + "geo": { + "alt": 12, + "lat": 50.962097, + "lon": 1.954764 + }, + "icao": "LFAC", + "id": 1254, + "type": "airport", + "tz": "Europe/Paris" } }, { - "x": { - "d": 4 + "travel-sample": { + "airportname": "Peronne St Quentin", + "city": "Peronne", + "country": "France", + "faa": null, + "geo": { + "alt": 295, + "lat": 49.868547, + "lon": 3.029578 + }, + "icao": "LFAG", + "id": 1255, + "type": "airport", + "tz": "Europe/Paris" } } -] - - - -

    +]
    +
    + ( select-expr ) +

    Use parenthesis to specify a N1QL SELECT expression of input + objects.

    +

    Arguments

    +

    + +
    select-expr
    +
    [Required] The N1QL SELECT query of input objects.
    +     +
    Example 1: A SELECT clause inside a + FROM clause.

    +

    List all Gillingham landmark names from a subset of all landmark + names and + addresses.SELECT name, city +FROM (SELECT id, name, address, city + FROM `travel-sample` + WHERE type = "landmark") as Landmark_Info +WHERE city = "Gillingham";Results:[ + { + "city": "Gillingham", + "name": "Royal Engineers Museum" + }, + { + "city": "Gillingham", + "name": "Hollywood Bowl" + }, + { + "city": "Gillingham", + "name": "Thai Won Mien" + }, + { + "city": "Gillingham", + "name": "Spice Court" + }, + { + "city": "Gillingham", + "name": "Beijing Inn" + }, + { + "city": "Gillingham", + "name": "Ossie's Fish and Chips" + } +]For + more details and examples, see SELECT Clause.

    - - -
    USE KEYS Clause -

    Specific primary keys within a keyspace can be specified. Only values having those primary - keys will be included as inputs to the query.

    -

    To specify a single key:

    - SELECT * FROM customer USE KEYS "acme-uuid-1234-5678" -

    To specify multiple keys:

    - SELECT * FROM customer USE KEYS [ "acme-uuid-1234-5678", "roadster-uuid-4321-8765" ] -

    In the FROM clause of a subquery, USE KEYS is - mandatory for the primary keyspace.

    -
    -
    JOIN Clause -

    The JOIN clause enables you to create new input objects by combining two or - more source objects. N1QL supports two types of joins: lookup joins and - index joins.

    -

    Lookup Joins

    Lookup joins allow only left-to-right joins, which means each qualified document from the - left-hand side (LHS) of the JOIN operator is required to produce - primary keys of documents on the right-hand side (RHS). These keys are subsequently - nested-loop-joined to access qualified RHS documents. Couchbase Server version 4.1 - and earlier supported only lookup joins.

    -

    The joins-clause is optional, and follows the FROM clause; it - allows you to combine two or more source objects to use as input objects. The - KEYS clause is required after each JOIN. It - specifies the primary keys of the documents for the second keyspace in the join.

    -

    Here is the syntax for the lookup join clause:

    - -[ join-type ] JOIN from-path [ [ AS ] alias ] on-keys-clause -

    where join-type [ LEFT ] is [ INNER | OUTER ] and from-path - is as discussed in the "from-path" section.

    -

    Lookup joins can be chained. By default, an INNER JOIN is performed. This means - that for each joined object produced, both the left-hand and right-hand source - objects must be non-missing and non-null.

    If LEFT or LEFT OUTER is specified, then a left outer join is - performed. At least one joined object is produced for each left-hand source object. - If the right-hand source object is NULL or MISSING, then the joined object's - right-hand side value is also NULL or MISSING (omitted), respectively.

    - -

    For example, if our customer objects were: { - "name": ..., - "primary_contact": ..., - "address": [ ... ] - }

    - -

    And our invoice objects were: { - "customer_key": ..., - "invoice_date": ..., - "invoice_item_keys": [ ... ], - "total": ... - }

    - -

    And the FROM clause was: FROM invoice inv JOIN customer cust ON KEYS inv.customer_key

    - -

    Then each joined object would - be: { - "inv" : { - "customer_key": ..., - "invoice_date": ..., - "invoice_item_keys": [ ... ], - "total": ... - }, - "cust" : { - "name": ..., - "primary_contact": ..., - "address": [ ... ] - } - }

    - -

    If our invoice_item objects were: - { - "invoice_key": ..., - "product_key": ..., - "unit_price": ..., - "quantity": ..., - "item_subtotal": ... - }

    - -

    And the FROM clause was: FROM invoice JOIN invoice_item item ON KEYS invoice.invoice_item_keys

    - -

    Then our joined objects would be: { - "invoice" : { - "customer_key": ..., - "invoice_date": ..., - "invoice_item_keys": [ ... ], - "total": ... - }, - "item" : { - "invoice_key": ..., - "product_key": ..., - "unit_price": ..., - "quantity": ..., - "item_subtotal": ... - } - }, - { - "invoice" : { - "customer_key": ..., - "invoice_date": ..., - "invoice_item_keys": [ ... ], - "total": ... - }, - "item" : { - "invoice_key": ..., - "product_key": ..., - "unit_price": ..., - "quantity": ..., - "item_subtotal": ... - } - }, - ...

    +
    + from-term +

    The from-term defines the input object(s) for the query, and it can be one of the + following types: + + + + + + Type + Example + + + + + keyspace identifier + `travel-sample` + + + generic + expression + 20+10 AS Total + + + subquery + SELECT t1.country, ARRAY_AGG(t1.city), + SUM(t1.city_cnt) AS apnum

    FROM + (SELECT city, city_cnt, + ARRAY_AGG(airportname) AS apnames, + country

    + FROM `travel-sample`

    + WHERE type = "airport"

    + GROUP BY city, country

    + LETTING city_cnt = COUNT(city) ) AS + t1

    WHERE t1.city_cnt > + 5;

     +     + + previous join, nest, or unnest + SELECT *

    FROM `travel-sample` AS + rte

    + JOIN `travel-sample` AS + aln

    ON rte.airlineid = + META(aln).id

    + NEST `travel-sample` AS + lmk

    ON aln.landmarkid = + META(lmk).id;

     +     + + +
    For more details with examples, click the above links.

    +

    + Couchbase Server version 4.6.2 adds support for generic expression in the + from-term. Prior Couchbase Server versions support only the other two + types. +

    +
    +
    ANSI JOIN Clause

    (Introduced in + Couchbase Server Enterprise Edition 5.5)

    ANSI JOIN (and ANSI + NEST) clauses have much more flexible functionality than their + earlier INDEX and LOOKUP equivalents. Since these are standard compliant and + more flexible, we recommend you to use ANSI JOIN (and ANSI NEST) exclusively, + where possible.Purpose

    To be closer to standard SQL syntax, + ANSI JOIN can join arbitrary fields of the documents and can be chained + together.

    The following table lists the JOIN types currently supported. + + + + + + + Join Type + Remarks + Example + + + + + [INNER] JOIN ... ON + INNER JOIN and LEFT OUTER JOIN can be mixed in + any number and/or order. + SELECT *

    FROM `travel-sample` + r

    JOIN + `travel-sample` + a

    ON + r.airlineid = META(a).id

    WHERE + a.country = "France"

     +     + + LEFT [OUTER] JOIN ... ON + SELECT *

    FROM `travel-sample` + r

    LEFT + JOIN `travel-sample` + a

    ON + r.airlineid = META(a).id

    WHERE + r.sourceairport = "SFO"

     +     + + RIGHT [OUTER] JOIN ... ON + RIGHT OUTER JOIN can only be the first join specified in a + FROM clause. + SELECT *

    FROM `travel-sample` + r

    RIGHT + JOIN `travel-sample` + a

    ON + r.airlineid = META(a).id

    WHERE + r.sourceairport = "SFO"

     +     + + +
    Syntax

    lhs-expr [join-type] JOIN rhs-expr ON join-clauseArguments

    + +
    lhs-expr
    +
    [Required] Keyspace reference or expression representing the left-hand + side of the join clause.
    +     +
    + +
    join-type
    +
    [Optional. Default is INNER] String representing the + type of join.
    +
    +
    + +
    INNER
    +
    [Optional. Default is INNER]
    +
    For each joined object produced, both the left-hand side and + right-hand side source objects of the ON + clause must be non-MISSING and non-NULL.
    +     +
    +
    + +
    LEFT [OUTER]
    +
    [Optional. Query Service interprets LEFT as + LEFT OUTER]
    +
    For each joined object produced, only the left-hand source + objects of the ON clause must be + non-MISSING and non-NULL
    +     +
    +
    + +
    RIGHT [OUTER]
    +
    [Optional. Query Service interprets RIGHT + as RIGHT OUTER]
    +
    For each joined object produced, only the right-hand source + objects of the ON clause must be + non-MISSING and non-NULL
    +     +
    +
    +     +
    + +
    JOIN rhs-expr
    +
    [Required] Keyspace reference or expression representing the right-hand + side of the join clause.
    +     +
    + +
    ON join-clause
    +
    [Required] Boolean expression representing the join condition between + the left-hand side expression and the right-hand side expression, which + can be fields, constant expressions or any complex N1QL expression.
    +     +
    ANSI Join Example 1: Inner Join.

    List the source airports and + airlines that fly into SFO, where only the non-null route documents + join with matching airline + documents.SELECT route.airlineid, airline.name, route.sourceairport, route.destinationairport +FROM `travel-sample` route +INNER JOIN `travel-sample` airline +ON route.airlineid = META(airline).id +WHERE route.type = "route" +AND route.destinationairport = "SFO" +ORDER BY route.sourceairport;Results:[ + { + "airlineid": "airline_5209", + "destinationairport": "SFO", + "name": "United Airlines", + "sourceairport": "ABQ" + }, + { + "airlineid": "airline_5209", + "destinationairport": "SFO", + "name": "United Airlines", + "sourceairport": "ACV" + }, + { + "airlineid": "airline_5209", + "destinationairport": "SFO", + "name": "United Airlines", + "sourceairport": "AKL" + }, +...ANSI + Join Example 2: Left Outer Join of U.S. airports in the same city as a + landmark.

    List the airports and landmarks in the same city, ordered by the airports. + SELECT DISTINCT MIN(aport.airportname) AS Airport__Name, + MIN(lmark.name) AS Landmark_Name, + MIN(aport.tz) AS Landmark_Time +FROM `travel-sample` aport +LEFT JOIN `travel-sample` lmark + ON aport.city = lmark.city + AND lmark.country = "United States" + AND lmark.type = "landmark" +WHERE aport.type = "airport" +GROUP BY lmark.name +ORDER BY lmark.name;

    Results:[ + { + "Airport__Name": "San Francisco Intl", + "Landmark_Name": "&quot;Hippie Temptation&quot; house", + "Landmark_Time": "America/Los_Angeles" + }, + { + "Airport__Name": "Los Angeles Intl", + "Landmark_Name": "101 Coffee Shop", + "Landmark_Time": "America/Los_Angeles" + }, + { + "Airport__Name": "San Francisco Intl", + "Landmark_Name": "1015", + "Landmark_Time": "America/Los_Angeles" + }, + { + "Airport__Name": "San Francisco Intl", + "Landmark_Name": "1235 Masonic Ave", + "Landmark_Time": "America/Los_Angeles" + }, +...ANSI + Join Example 3: RIGHT OUTER JOIN of Example #2.

    List the airports and landmarks in the same city, ordered by the landmarks.The LEFT OUTER + JOIN will list all left-side results regardless of matching right-side + documents; while the RIGHT OUTER JOIN will list all right-side results + regardless of matching left-side + documents.SELECT DISTINCT MIN(aport.airportname) AS Airport_Name, + MIN(lmark.name) AS Landmark_Name, + MIN(aport.tz) AS Landmark_Time +FROM `travel-sample` aport +RIGHT JOIN `travel-sample` lmark + ON aport.city = lmark.city + AND aport.type = "airport" + AND aport.country = "United States" +WHERE lmark.type = "landmark" +GROUP BY lmark.name +ORDER BY lmark.name;Results:[ + { + "Airport_Name": "San Francisco Intl", + "Landmark_Name": "&quot;Hippie Temptation&quot; house", + "Landmark_Time": "America/Los_Angeles" + }, + { + "Airport_Name": "London-Corbin Airport-MaGee Field", + "Landmark_Name": "02 Shepherd's Bush Empire", + "Landmark_Time": "America/New_York" + }, + { + "Airport_Name": "Los Angeles Intl", + "Landmark_Name": "101 Coffee Shop", + "Landmark_Time": "America/Los_Angeles" + }, + { + "Airport_Name": "San Francisco Intl", + "Landmark_Name": "1015", + "Landmark_Time": "America/Los_Angeles" + }, +...ANSI + Join Example #4: In the `beer-sample` bucket, use an ANSI + JOIN to list the beer names and breweries that are in the state Wisconsin + (WI). First, create an index with + beer.brewry_id as the leading + key.CREATE INDEX beer_brewery ON `beer-sample` (brewery_id) +WHERE type = "beer" -

    ON KEYS is required after each JOIN. It specifies the - primary keys for the second keyspace in the join.

    -

    Joins can be chained.

    -

    By default, an INNER JOIN is performed. This means that for each joined - object produced, both the left and right hand source objects must be non-missing and - non-null.

    -

    If LEFT or LEFT OUTER is specified, then a left outer join - is performed. At least one joined object is produced for each left-hand source - object. If the right-hand source object is NULL or MISSING, then the joined object's - right-hand side value is also NULL or MISSING (omitted), respectively.

    - -

    Index Joins

    -

    When using lookup joins, right-to-left joins with RHS documents containing primary key - references to LHS documents cannot be joined efficiently using any index.

    -

    For example, consider the travel-sample data with route and airline documents, - where route.airlineid is the primary key of route documents and - airline documents have no reference to route documents. The following query to get a - list of flights from airlines flying to SFO cannot be efficiently executed without - making a Cartesian product of all route documents (LHS) with all airline documents - (RHS): +SELECT META(brewery).id bid, META(beer).id, brewery.name brewery_name, + beer.name beer_name +FROM `beer-sample` brewery +JOIN `beer-sample` beer + ON beer.brewery_id = LOWER(REPLACE(brewery.name, " ", "_")) +WHERE beer.type = "beer" + AND brewery.type = "brewery" + AND brewery.state = "WI";Results:[ + { + "beer_name": "Dank", + "bid": "oso", + "brewery_name": "Oso", + "id": "oso-dank" + } + ]Visual + Explain Plan:

    If you add name as the second + index key to the beer_brewery + index:CREATE INDEX beer_brewery_name ON `beer-sample` (brewery_id, name) +WHERE type = "beer"... + then you will get covering index scan, as shown in the Visual Explain + Plan:

    Limitations

    The following Join types are + currently not supported:

      +
    • RIGHT OUTER JOIN is only supported when it’s the only join + in the query; or in a chain of joins, the RIGHT OUTER JOIN + must be the first join in the chain.
      • +
    • No mixing of new ANSI Join syntax with Lookup/Index Join syntax in the same + FROM clause.
      • +
    • The right-hand-side of any join must be a keyspace. Expressions, subqueries, + or other join combinations cannot be on the right-hand-side of a join.
      • +
    • A join can only be executed when appropriate index exists on the inner side + of the join.
      • +
    • Adaptive indexes are not considered when selecting indexes on inner side of + the join.
      • +

    +
    + ANSI JOIN Hints (HASH & NL) +

    (Introduced in Couchbase Server Enterprise Edition 5.5)

    +

    Couchbase Server Enterprise Edition supports two join methods for performing ANSI + Join: nested-loop join and hash join. The default join method is nested-loop join. + Two corresponding join hints are introduced: USE HASH and + USE NL.

    +

    Hash join is only considered when the USE HASH hint is specified, + and it requires at least one equality predicate between the left-hand side and + right-hand side. In such cases, if a hash join is chosen successfully, then that’ll + be the join method used for this join. If the hash join cannot be generated, then + the planner will further consider nested-loop join and will either generate a + nested-loop join or return an error for the join.

    +

    If no join hint is specified or USE NL hint is specified, then nested-loop join is + considered.

    +

    + For Community Edition (CE), any specified USE HASH hint will + be silently ignored and only nested-loop join is considered by the + planner. +

    +

    USE HASH hint

    +

    The USE HASH hint is similar to the existing USE INDEX + or USE KEYS hint in that the USE HASH + hint can be specified after a keyspace reference in an ANSI Join + specification. There are two versions of the USE HASH hint that + indicate whether the keyspace is to be used as:

      +
    • The build side of the hash join -- USE HASH(build)
      • +
    • The probe side of the hash join -- USE HASH(probe)
      • +

    +

    A hash join has two sides: a BUILD and a PROBE. The + BUILD side of the join will be used to create an in-memory hash + table. The PROBE side will use that table to find matches and + perform the join. Typically, this means you want the BUILD side to + be used on the smaller of the two sets. However, you can only supply one hash hint, + and only to the right side of the join. So if you specify BUILD on + the right side, then you are implicitly using PROBE on the left + side (and vice versa).

    +

    USE HASH Example 1: PROBE

    +

    The keyspace aline is to be joined (with rte) using + hash join, and aline is used as the probe side of the hash + join.SELECT COUNT(1) AS Total_Count +FROM `travel-sample` rte +INNER JOIN `travel-sample` aline +USE HASH (PROBE) +ON (rte.airlineid = META(aline).id) +WHERE rte.type = "route";Results:[ + { + "Total_Count": 17629 + } +]USE + HASH Example 2: BUILD

    +

    This is effectively the same query as the previous example, except the two keyspaces + are switched, and here the USE HASH(BUILD) hint is used, indicating + the hash join should use rte as the build + side.SELECT COUNT(1) AS Total_Count +FROM `travel-sample` aline +INNER JOIN `travel-sample` rte +USE HASH (BUILD) +ON (rte.airlineid = META(aline).id) +WHERE rte.type = "route";Results:[ + { + "Total_Count": 17629 + } +]USE + NL hint

    +

    This join hint instructs the planner to use nested-loop join (NL join) for the join + being considered. Since nested-loop join is the default path, the USE + NL hint is not required.

    +

    USE NL + Example:SELECT COUNT(1) AS Total_Count +FROM `travel-sample` rte +INNER JOIN `travel-sample` aline +USE NL +ON (rte.airlineid = META(aline).id) +WHERE rte.type = "route";The + join hint for the first join should be specified on the 2nd keyspace reference, + and the join hint for the second join should be specified on the 3rd keyspace + reference, etc. If a join hint is specified on the first keyspace, an error is + returned.Multiple hints

    +

    You can use only one join hint (USE HASH or USE NL) together with only one other hint + (USE INDEX or USE KEYS) for a total of two hints. The order of the two hints doesn't + matter.

    +

    When multiple hints are being specified, use only one USE keyword + with one following the other, as in the following examples.

    +

    Multiple hint Example 1: USE INDEX with USE HASH.

    +

    SELECT COUNT(1) AS Total_Count +FROM `travel-sample` rte +INNER JOIN `travel-sample` aline +USE INDEX idx1 HASH (PROBE) +ON (rte.airlineid = META(aline).id) +WHERE rte.type = "route";Multiple + hint Example 2: USE HASH with USE + KEYS.SELECT COUNT(1) AS Total_Count +FROM `travel-sample` rte +INNER JOIN `travel-sample` aline +USE HASH (PROBE) KEYS ["airline_key1", "airline_key2", "airline_key3"] +ON (rte.airlineid = META(aline).id) +WHERE rte.type = "route";When + chosen, the hash join will always work; the restrictions are on any USE KEYS hint + clause:

      +
    • Must not depend on any previous keyspaces.
      • +
    • The expression must be constants, host variables, etc.
      • +
    • Must not contain any subqueries.
      • +
    If the USE KEYS hint contains references to other keyspaces or + subqueries, then the USE HASH hint will be ignored and nested-loop join will be + used instead.

    +
    +
    + ANSI JOIN and Arrays +

    ANSI JOIN provides great flexibility since the ON clause of an ANSI + JOIN can be any expression as long as it evaluates to TRUE or FALSE. Below are + different join scenarios involving arrays and ways to handle each + scenario.These buckets and indexes will be used throughout this section's + array scenarios. As a convention, when a field name starts with a + it is an array, so each bucket has two array fields and two regular + fields. Also, both _idx1 indexes index each element of its + array, while both _idx2 indexes use its entire array as the + index key.

    bucket b1 (a11, a12, c11, c12)

    bucket + b2 (a21, a22, c21, c22)

    CREATE INDEX b1_idx1 ON b1 (c11, + c12, DISTINCT a11)

    CREATE INDEX b1_idx2 ON b1 + (a12)

    CREATE INDEX b2_idx1 ON b2 (c21, c22, DISTINCT + a21)

    CREATE INDEX b2_idx2 ON b2 + (a22)

    ANSI JOIN with no arrays

    +

    In this scenario, there is no involvement of arrays in the join. These are just + straight-forward + joins:SELECT * +FROM b1 +JOIN b2 + ON b1.c11 = b2.c21 + AND b2.c22 = 100 +WHERE b1.c12 = 10;Here + the joins are using non-array fields of each keyspace.

    +

    The following case also falls in this + scenario:SELECT * +FROM b1 +JOIN b2 + ON b1.c11 = b2.c21 + AND b2.c22 = 100 + AND ANY v IN b2.a21 SATISFIES v = 10 END +WHERE b1.c12 = 10;In + this example, although there is an ANY predicate on the right-hand side array + b2.a21, the ANY predicate does not involve any joins, and thus, + as far as the join is concerned, it is still a 1-to-1 join. + Similarly:SELECT * +FROM b1 +JOIN b2 + ON b1.c11 = b2.c21 +WHERE b1.c11 = 10 + AND b1.c12 = 100 + AND ANY v IN b1.a11 SATISFIES v = 20 END;In + this case the ANY predicate is on the left-hand side array b1.a11; + however, similar to above, the ANY predicate does not involve any joins, and thus + the join is still 1-to-1. We can even have ANY predicates on both + sides:SELECT * +FROM b1 +JOIN b2 + ON b1.c11 = b2.c21 + AND b2.c22 = 100 + AND ANY v IN b2.a21 SATISFIES v = 10 END +WHERE b1.c11 = 10 + AND b1.c12 = 100 + AND ANY v IN b1.a11 SATISFIES v = 10 END;Again, + the ANY predicates do not involve any join, and the join is still 1-to-1.

    +

    ANSI JOIN with entire array as index key

    +

    As a special case, it is possible to perform ANSI JOIN on an entire array as a join + key:SELECT * +FROM b1 +JOIN b2 + ON b1.a21 = b2.a22 +WHERE b1.c11 = 10 + AND b1.c12 = 100;In + this case, the entire array must match each other for the join to work. For all + practical purposes, the array here is treated as a scalar since there is no logic to + iterate through elements of an array here. The entire array is used as an index key + (b2_idx2) and as such, an entire array is used as an index span + to probe the index. The join here can also be considered as 1-to-1.

    +

    ANSI JOIN involving right-hand-side arrays

    +

    In this scenario, the join involves an array on the right-hand side + keyspace:SELECT * +FROM b1 +JOIN b2 + ON b2.c21 = 10 + AND b2.c22 = 100 + AND ANY v IN b2.a21 SATISFIES v = b1.c12 END +WHERE b1.c11 = 10;In + this case, the ANY predicate involves a join, and thus, effectively we are joining + b1 with elements of the b2.a21 array. This now + becomes a 1-to-many join. Note that we use an ANY clause for this scenario since + it’s a natural extension of the existing support for array indexes; the only + difference is for index span generation, we now can have a potential join + expression. Array indexes can be used for join in this scenario.

    +

    ANSI JOIN involving left-hand-side arrays

    +

    This is a slightly more complex scenario, where the array reference is on the + left-hand side of the join, and it’s a many-to-1 join. There are two alternative + ways to handle the scenario where the array appears on the left-hand side of the + join.

    +

    +

    + +
    Alternative #1: use UNNEST
    +
    This alternative will flatten the left-hand side array first, before + performing the + join:SELECT * +FROM b1 UNNEST b1.a12 AS ba1 +JOIN b2 + ON ba1 = b2.c22 + AND b2.c21 = 10 +WHERE b1.c11 = 10 + AND b1.c12 = 100;The + UNNEST + operation is used to flatten the array, turning one left-hand side + document into multiple documents; and then for each one of them, join + with the right-hand side. This way, by the time join is being performed, + it is a regular join, since the array is already flattened in the UNNEST + step.
    +     +
    +
    + +
    Alternative #2: use IN clause
    +
    This alternative uses the IN clause to handle the + array:SELECT * +FROM b1 +JOIN b2 + ON b2.c22 IN b1.a12 AND b2.c21 = 10 +WHERE b1.c11 = 10 AND b1.c12 = 100;By + using the IN + clause, the right-hand side field value can match any of the elements of + the left-hand side array. Conceptually, we are using each element of the + left-hand side array to probe the right-hand side index.
    +     +
    +
    + +
    Differences between the two alternatives
    +
    There is a semantical difference between the two alternatives. With + UNNEST, we are first turning one left-hand side document into multiple + documents and then performing the join. With IN-clause, there is still + only one left-hand side document, which can then join with one or more + right-hand side documents. Thus:
      +
    • If the array contains duplicate values,
        +
      • the UNNEST method treats each duplicate as an individual + value and thus duplicated results will be returned;
        • +
      • the IN clause method will not duplicate the result. +
        • +
      • +
    • If no duplicate values exists and we are performing inner join, +
        +
      • then the two alternatives will likely give the same + result.
        • +
      • +
    • If outer join is performed, assuming there are N elements in the + left-hand side array, and assuming there is at most one matching + document from the right-hand side for each element of the array, +
        +
      • the UNNEST method will produce N result documents;
        • +
      • the IN clause method may produce < N result documents + if some of the array elements do not have matching + right-hand side documents.
        • +
      • +
    +     +
    +

    +

    ANSI JOIN with arrays on both sides

    +

    If the join involves arrays on both sides, then we can combine the approaches above, + i.e., using ANY clause to handle the right-hand side array and either UNNEST or IN + clause to handle the left-hand side array. For + example:SELECT * +FROM b1 +UNNEST b1.a12 AS ba1 + JOIN b2 + ON ANY v IN b2.a21 SATISFIES v = ba1 END + AND b2.c21 = 10 + AND b2.c22 = 100 +WHERE b1.c11 = 10 + AND b1.c12 = 100;orSELECT * +FROM b1 +JOIN b2 + ON ANY v IN b2.a21 SATISFIES v IN b1.a12 END + AND b2.c21 = 10 + AND b2.c22 = 100 +WHERE b1.c11 = 10 + AND b1.c12 = 100;

    +
    +
    + Lookup JOIN Clause +

    (Introduced in Couchbase Server 4.0)

    +

    The JOIN clause enables you to create new input objects by combining + two or more source objects.

    +

    Lookup joins allow only left-to-right joins, which means the ON KEYS expression must + produce a document key which is then used to retrieve documents from the right-hand + side keyspace. Couchbase Server version 4.1 and earlier supported only lookup + joins.

    +

    Syntax

    +

    + [ join-type ] JOIN from-path [ [ AS ] alias ] ON KEYS on-keys-clause +

    +

    + +
    Arguments
    +
    +
    + +
    join-type
    +
    [Optional; default is INNER]
    +
    +
    + +
    INNER
    +
    For each joined object produced, both the + left-hand and right-hand source objects must be + non-MISSING and + non-NULL.
    +     +
    +
    + +
    LEFT OUTER
    +
    For each joined object produced, only the + left-hand source objects must be + non-MISSING and + non-NULL.
    +     +
    +
    +     +
    +
    + +
    from-path
    +
    [Required] Keyspace reference for right-hand side of lookup + join. For details, see Keyspaces.
    +     +
    +
    + +
    alias (Optionally, AS alias)
    +
    [Optional] To assign another name. For details, see AS + Keyword.
    +     +
    +
    + +
    ON KEYS on-keys-clause
    +
    [Required] String or expression representing the primary + keys of the documents for the right-hand side keyspace.
    +
    The ON KEYS expression produces one or more + document keys for the right-hand side document.
    +
    The ON KEYS expression can produce an array + of document keys.
    +     +
    +
    +     +
    +
    + +
    Return Values
    +
    If LEFT or LEFT OUTER is specified, + then a left outer join is performed.
    +
    At least one joined object is produced for each left-hand source + object.
    +
    If the right-hand source object is NULL or + MISSING, then the joined object's right-hand side + value is also NULL or MISSING + (omitted), respectively.
    +     +
    +
    + +
    Limitations
    +
    Lookup JOINs can be chained with other lookup joins/nests or index + joins/nests, but they cannot be mixed with an ANSI JOIN or ANSI + NEST.
    +     +
    +

    +

    + Lookup JOIN Example 1: route JOIN airline ON KEYS route.airlineid.

    +

    List all airlines and non-stop routes from SFO in the travel-sample + keyspace. SELECT DISTINCT airline.name, airline.callsign, route.destinationairport, route.stops, route.airline +FROM `travel-sample` route + JOIN `travel-sample` airline + ON KEYS route.airlineid +WHERE route.type = "route" +AND airline.type = "airline" +AND route.sourceairport = "SFO" +AND route.stops = 0 +LIMIT 4;

    +

    Results:[ + { + "airline": "VX", + "callsign": "REDWOOD", + "destinationairport": "SAN", + "name": "Virgin America", + "stops": 0 + }, + { + "airline": "VX", + "callsign": "REDWOOD", + "destinationairport": "PHL", + "name": "Virgin America", + "stops": 0 + }, + { + "airline": "B6", + "callsign": "JETBLUE", + "destinationairport": "FLL", + "name": "JetBlue Airways", + "stops": 0 + }, + { + "airline": "UA", + "callsign": "UNITED", + "destinationairport": "IND", + "name": "United Airlines", + "stops": 0 + } +]

    +

    Lookup JOIN Example 2: route JOIN airline ON KEYS route.airlineid.

    +

    List the schedule of flights from Boston to San Francisco on JETBLUE in the + travel-sample + keyspace.SELECT DISTINCT airline.name, route.schedule +FROM `travel-sample` route + JOIN `travel-sample` airline + ON KEYS route.airlineid +WHERE route.type = "route" +AND airline.type = "airline" +AND route.sourceairport = "BOS" +AND route.destinationairport = "SFO" +AND airline.callsign = "JETBLUE";

    +

    Results:[ + { + "name": "JetBlue Airways", + "schedule": [ + { + "day": 0, + "flight": "B6076", + "utc": "10:15:00" + }, + { + "day": 0, + "flight": "B6321", + "utc": "00:06:00" + }, + { + "day": 1, + "flight": "B6536", + "utc": "22:45:00" + }, + { + "day": 1, + "flight": "B6194", + "utc": "00:51:00" + }, + { + "day": 2, + "flight": "B6918", + "utc": "23:45:00" + }, + { + "day": 2, + "flight": "B6451", + "utc": "18:09:00" + }, + { + "day": 2, + "flight": "B6868", + "utc": "22:04:00" + }, + { + "day": 2, + "flight": "B6621", + "utc": "11:04:00" + }, + { + "day": 3, + "flight": "B6015", + "utc": "16:59:00" + }, + { + "day": 3, + "flight": "B6668", + "utc": "07:22:00" + }, + { + "day": 3, + "flight": "B6188", + "utc": "01:41:00" + }, + { + "day": 3, + "flight": "B6215", + "utc": "19:35:00" + }, + { + "day": 4, + "flight": "B6371", + "utc": "21:37:00" + }, + { + "day": 4, + "flight": "B6024", + "utc": "10:24:00" + }, + { + "day": 4, + "flight": "B6749", + "utc": "01:12:00" + }, + { + "day": 4, + "flight": "B6170", + "utc": "01:14:00" + }, + { + "day": 5, + "flight": "B6613", + "utc": "08:59:00" + }, + { + "day": 5, + "flight": "B6761", + "utc": "15:24:00" + }, + { + "day": 5, + "flight": "B6162", + "utc": "02:42:00" + }, + { + "day": 5, + "flight": "B6341", + "utc": "21:26:00" + }, + { + "day": 5, + "flight": "B6347", + "utc": "08:43:00" + }, + { + "day": 6, + "flight": "B6481", + "utc": "22:08:00" + }, + { + "day": 6, + "flight": "B6549", + "utc": "21:48:00" + }, + { + "day": 6, + "flight": "B6994", + "utc": "11:30:00" + }, + { + "day": 6, + "flight": "B6892", + "utc": "13:27:00" + } + ] + } +]

    +

    +

    +
    Index JOIN Clause

    (Introduced in + Couchbase Server 4.0)

    When Lookup JOINs cannot efficiently join + left-hand side documents with right-to-left joins and your situation cannot be + flipped because your predicate needs to be on the left-hand side (such as the above + Lookup Example #1 where airline documents have no reference to route documents), + then Index JOINs can be used efficiently without making a Cartesian product of all + route documents. Index JOINs allow you to flip the direction of your join + clause.

    Consider the below query similar to the above Lookup Example #1 + with route and airline documents where route.airlineid is the document key of route + documents and airline documents have no reference to route + documents:SELECT DISTINCT airline.name, airline.callsign, route.destinationairport, + route.stops, route.airline FROM `travel-sample` route JOIN `travel-sample` airline ON KEYS route.airlineid WHERE route.type = "route" - AND airline.type = "airline" - AND route.sourceairport = "SFO" -LIMIT 2;This - query cannot use any index on airline to directly access - sourceairport in SFO because airline is on - the RHS. Also, you cannot rewrite the query to put the airline document on the LHS - (to use any index) and the route document on the RHS because the - airline documents (on the LHS) have no primary keys to access - the route documents (on the RHS).

    -

    Using the new index joins, the same query can be written as: +AND airline.type = "airline" +AND airline.icao = "SEA" +LIMIT 4;This + query gets a list of Seattle (SEA) flights, but getting + SEA flights cannot be efficiently executed without making a + Cartesian product of all route documents (LHS) with all airline documents + (RHS).

    This query cannot use any index on airline to directly access SEA + flights because airline is on the RHS.

    Also, you cannot rewrite the query to + put the airline document on the LHS (to use any index) and the route document on the + RHS because the airline documents (on the LHS) have no primary keys to access the + route documents (on the RHS).

    Using index joins, the same query can be written + as: Required Index: -CREATE INDEX route.airlineid ON `travel-sample`(airlineid) WHERE type="route"; + CREATE INDEX route_airlineid ON `travel-sample`(airlineid) WHERE type="route"; Optional index: -CREATE INDEX route.sourceairport ON `travel-sample`(sourceairport) WHERE type="route"; + CREATE INDEX airline_icao ON `travel-sample`(icao) WHERE type="airline"; Resulting in: -SELECT * FROM `travel-sample` airline JOIN `travel-sample` route + SELECT * FROM `travel-sample` airline + JOIN `travel-sample` route ON KEY route.airlineid FOR airline - WHERE route.type="route" AND airline.type="airline" AND route.sourceairport = "SFO";

    -

    If you generalize the same query, it looks like the following: + WHERE route.type="route" + AND airline.type="airline" + AND airline.icao = "SEA";

    If + you generalize the same query, it looks like the following: CREATE INDEX on-key-for-index-name rhs-expression (lhs-expression-key); -SELECT projection-list FROM lhs-expression JOIN rhs-expression - ON KEY rhs-expression.lhs-expression-key FOR lhs-expression - [ WHERE predicates ] ;

    -

    There are three important changes in the index scan syntax example above:

      projection-list +FROM lhs-expression +JOIN rhs-expression + ON KEY rhs-expression.lhs-expression-key FOR lhs-expression +[ WHERE predicates ] ;

      There + are three important changes in the index scan syntax example above:

      • CREATE INDEX on the ON KEY expression route.airlineid to access route @@ -499,246 +1491,1184 @@ SELECT projection-list FROM lhs-expression LHS).
      • The ON KEY route.airlineid FOR airline enables N1QL to use the index route.airlineid.
        • -
      • Create any optional index such as route.sourceairport that - can be used on route (LHS).
        • +
      • Create any optional index such as route.airline that can be + used on airline (LHS).
      For index joins, the syntax uses ON KEY (singular) - instead of ON KEYS (plural). This is because for index joins, - the ON KEY expression must produce a scalar value. For lookup - joins, the ON KEYS expression can produce either a scalar or an - array value.

      -

      Syntax of Index Joins

      -

      A new on-key-for-clause is added to the join-clause of the - FROM syntax.

      - [ join-type ] JOIN from-path [ [ AS ] alias ] < on-key-for-clause > -

      on-keys-clause:

      - ON [ PRIMARY ] KEYS expression -

      on-key-for-clause

      - ON [ PRIMARY ] KEY rhs-expression.lhs-expression-key FOR lhs-expression -

      rhs-expression: keyspace or expression corresponding to the right hand side of JOIN.

      -

      lhs-expression: keyspace or expression corresponding to the left hand side of JOIN.

      -

      lhs-expression-key: attribute in rhs-expression - referencing primary key for lhs-expression.

      -

      Example

      -

      The following example counts the number of distinct "AA" airline routes for each - airport:

      - SELECT Count(DISTINCT route.sourceairport) AS distinctairports1 -FROM `travel-sample` airline + instead of ON KEYS (plural). This is because Index JOINs' + ON KEY expression must produce a scalar value; whereas + Lookup JOINs' ON KEYS expression can produce either a scalar or + an array value.

      Syntax

      [ join-type ] JOIN from-path [ [ AS ] alias ] ON KEY FOR on-key-for-clause

      +

      + +
      Arguments
      +
      +
      + +
      join-type
      +
      [Optional; default is LEFT INNER]
      +
      +
      + +
      LEFT or LEFT + INNER
      +
      For each joined object produced, both the + left-hand and right-hand source objects must be + non-MISSING and + non-NULL.
      +       +
      +
      + +
      LEFT OUTER
      +
      For each joined object produced, only the + left-hand source objects must be + non-MISSING and + non-NULL.
      +       +
      +
      +       +
      +
      + +
      from-path
      +
      Keyspace reference for right-hand side of an index join. For + details, see Keyspaces.
      +       +
      +
      + +
      AS alias
      +
      [Optional] To assign another name. For details, see AS + Keyword.
      +       +
      +
      + +
      ON KEY rhs-expression.lhs-expression-key
      +
      +
      + +
      rhs-expression
      +
      Keyspace reference for the right-hand side of + the index join.
      +       +
      +
      + +
      lhs-expression-key
      +
      String or expression representing the attribute + in rhs-expression referencing the document key for + lhs-expression.
      +       +
      +
      +       +
      +
      + +
      FOR lhs-expression
      +
      Keyspace reference for the left-hand side of the index + join.
      +       +
      +
      +       +
      +

      Index JOIN Example 1: + ON KEY ... FOR.

      The following example counts the number of + distinct "AA" airline routes for each airport after creating the following index (if + not already + created).

      CREATE INDEX route_airlineid ON `travel-sample`(airlineid) WHERE type="route"; + +SELECT Count(DISTINCT route.sourceairport) AS DistinctAirports +FROM `travel-sample` airline JOIN `travel-sample` route - ON KEY route.airlineid FOR airline -WHERE route.type = "route" - AND airline.type = "airline" - AND airline.iata = "AA"; - -
    -
    UNNEST Clause -

    If a document or object contains a nested array, UNNEST conceptually performs - a join of the nested array with its parent object. Each resulting joined object - becomes an input to the query. Unnests can be chained.

    -

    Here is the syntax for an UNNEST join:

    -[ join-type ] UNNEST path [ [ AS ] alias ] -

    where join-type is [ INNER | LEFT [ OUTER ] ]

    -

    The first path element after each UNNEST must reference some preceding path.

    -

    By default, an INNER UNNEST is performed. This means that for each result - object produced, both the left-hand and right-hand source objects must be - non-missing and non-null.

    -

    If LEFT or LEFT OUTER is specified, then a left outer unnest - is performed. At least one result object is produced for each left source object. If - the right-hand source object is NULL, MISSING, - empty, or a non-array value, then the result object's right-side value is - MISSING (omitted).

    -

    Example

    -

    If some customer documents contain an array of addresses under the address field, the following query retrieves - each nested address along with the parent customer's name.

    -SELECT c.name, a.* FROM customer c UNNEST c.address a -

    Here is the result set:

    - [ - { "name" : "Acme Inc.", "street" : "101 Main St.", - "zip" : "94040" }, - { "name" : "Acme Inc.", "street" : "300 Broadway", - "zip" : "10011" }, - { "name" : "Roadster Corp.", "street" : "3500 Wilshire Blvd.", - "zip" : "90210" }, - { "name" : "Roadster Corp.", "street" : "4120 Alamo Dr.", - "zip" : "75019" } -] -

    In the following example, The UNNEST clause iterates over the reviews array - and collects the reviewerName and publication from - each element in the array. This collection of objects can be used as input for other - query operations.

    -SELECT review.reviewerName, review.publication - FROM beers AS b - UNNEST review IN b.reviews -

    Here is the result set:

    -{"id": "7983345", - "name": "Takayama Pale Ale", - "brewer": "Hida Takayama Brewing Corp.", - "reviews" : [ - {"reviewerName" : "Takeshi Kitano", - "publication" : "Outdoor Japan Magazine","date": "3/2013"}, - {"reviewerName" : "Moto Ohtake", "publication" : "Japan Beer Times", - "date" : "7/2013"} - ] -} -
    -
    NEST Clause - [ join-type ] NEST from-path [ [ AS ] alias ] on-keys-clause -

    where join-type is [ INNER | LEFT [ OUTER ] ]

    -

    Nesting is conceptually the inverse of unnesting. Nesting performs a join across two keyspaces. But - instead of producing a cross-product of the left and right inputs, a single result is produced for each left - input, while the corresponding right inputs are collected into an array and nested as a single array-valued field - in the result object.

    -

    Nests can be chained with other NEST, JOIN, and - UNNEST clauses. By default, an INNER NEST is - performed. This means that for each result object produced, both the left and right - source objects must be non-missing and non-null. The right-hand side result of - NEST is always an array or MISSING. If there - is no matching right source object, then the right source object is as - follows: - + ON KEY route.airlineid FOR airline +WHERE route.type = "route" +AND airline.type = "airline" +AND airline.iata = "AA"; Results:[ + { + "DistinctAirports": 429 + } +] +
    UNNEST Clause

    If a document or object contains a + nested array, UNNEST conceptually performs a join of the nested array with its + parent object. Each resulting joined object becomes an output of the query. Unnests + can be chained.

    Syntax

    [ join-type ] UNNEST path [ [ AS ] alias ]

    +

    + +
    Arguments
    +
    +
    + +
    join-type
    +
    [Optional; default is INNER]
    +
    +
    + +
    INNER
    +
    For each result object produced, the array + object in the left-hand side keyspace must be + non-empty.
    +     +
    +
    + +
    LEFT or LEFT + OUTER
    +
    A left-outer unnest is performed, and at least + one result object is produced for each left source + object.
    +     +
    +
    +     +
    +
    + +
    path
    +
    [Required] The first path element after each UNNEST must + reference some preceding path.
    +     +
    +
    + +
    alias (optionally, AS alias)
    +
    [Required] To assign a name for the unnested item. For + details, see AS Keyword.
    +     +
    +
    +     +
    +
    + +
    Return Values
    +
    If the right-hand source object is NULL, + MISSING, empty, or a non-array value, then the + result object's right-side value is MISSING + (omitted).
    +     +
    +

    UNNEST Example 1: UNNEST an array to select an item.

    In the + `travel-sample` keyspace, flatten the schedule array to get a list of the flights on + Monday + (1).

    SELECT sched +FROM `travel-sample` +UNNEST schedule sched +WHERE sched.day = 1 +LIMIT 3;

    Results + :

    [ + { + "sched": { + "day": 1, + "flight": "AF356", + "utc": "12:40:00" + } + }, + { + "sched": { + "day": 1, + "flight": "AF480", + "utc": "08:58:00" + } + }, + { + "sched": { + "day": 1, + "flight": "AF250", + "utc": "12:59:00" + } + } +]Another + way to get similar results is by using a Collection Operator to find array items that + meet our + criteria:SELECT ARRAY item FOR item IN schedule WHEN item.day = 1 END AS Monday_flights +FROM `travel-sample` +WHERE type = "route" +AND ANY item IN schedule SATISFIES item.day = 1 END +LIMIT 3;However, + without the UNNEST clause, the unflattened list results in 3 sets of + flights instead of only 3 individual + flights:[ + { + "Monday_flights": [ + { + "day": 1, + "flight": "AF356", + "utc": "12:40:00" + }, + { + "day": 1, + "flight": "AF480", + "utc": "08:58:00" + }, + { + "day": 1, + "flight": "AF250", + "utc": "12:59:00" + }, + { + "day": 1, + "flight": "AF130", + "utc": "04:45:00" + } + ] + }, + { + "Monday_flights": [ + { + "day": 1, + "flight": "AF517", + "utc": "13:36:00" + }, + { + "day": 1, + "flight": "AF279", + "utc": "21:35:00" + }, + { + "day": 1, + "flight": "AF753", + "utc": "00:54:00" + }, + { + "day": 1, + "flight": "AF079", + "utc": "15:29:00" + }, + { + "day": 1, + "flight": "AF756", + "utc": "06:16:00" + } + ] + }, + { + "Monday_flights": [ + { + "day": 1, + "flight": "AF975", + "utc": "11:23:00" + }, + { + "day": 1, + "flight": "AF225", + "utc": "16:05:00" + } + ] + } +]

    UNNEST + Example 2: Use UNNEST to collect items from one array to + use in another query.

    In this example, the UNNEST clause + iterates over the reviews array and collects the + author names of the reviewers who rated the rooms less than a 2 + to be contacted for ways to improve. r is an element of the array + generated by the UNNEST + operation.

    SELECT RAW r.author +FROM `travel-sample` +UNNEST reviews AS r +WHERE `travel-sample`.type = "hotel" +AND r.ratings.Rooms < 2 +LIMIT 4;

    This + results + in:

    [ + "Kayli Cronin", + "Shanelle Streich", + "Catharine Funk", + "Tyson Beatty" +]
    +
    + ANSI NEST Clause +

    (Introduced in Couchbase Server Enterprise Edition 5.5)

    +

    ANSI NEST (and ANSI JOIN) clauses are much faster and have much more flexible + functionality than their earlier INDEX and LOOKUP equivalents, so users are + strongly recommended to use ANSI NEST (and ANSI JOIN) exclusively, where + possible.ANSI NEST supports more nest types than Couchbase Server version + 4.0's NEST was able. ANSI NEST can nest arbitrary fields of the documents and can be + chained together.

    +

    The key difference between the currently supported nests and ANSI NEST support is the + replacement of the current ON KEYS or ON KEY … FOR + clauses with a simple ON clause. The ON KEYS or + ON KEY … FOR clauses dictate that those nests can only be done + on a document key (primary key for a document). The ON clause can + contain any expression, and thus it opens up many more nest possibilities that + Couchbase did not previously support.

    +

    Syntax

    +

    +

    lhs-expr + [nest-type] NEST + rhs-expr ON + nest-clause

    +

    Arguments

    +

    + +
    lhs-expr
    +
    [Required] Keyspace reference or expression representing the left-hand + side of the nest clause.
    +     +
    + +
    nest-type
    +
    [Optional. Default is INNER] String representing the + type of nest.
    +
    +
    + +
    INNER
    +
    [Optional. Default is INNER]
    +
    For each nested object produced, both the left-hand and + right-hand source objects must be non-MISSING and + non-NULL.
    +     +
    +
    + +
    LEFT [OUTER]
    +
    [Optional. Query Service interprets LEFT as + LEFT OUTER]
    +
    For each nested object produced, only the left-hand source + objects must be non-MISSING and non-NULL.
    +     +
    +
    +     +
    + +
    NEST rhs-expr
    +
    [Required] Keyspace reference or expression representing the right-hand + side of the nest clause.
    +     +
    + +
    ON nest-clause
    +
    [Required] Boolean expression representing the nest condition between + the left-hand side expression and the right-hand side expression, which + can be fields, constant expressions or any complex N1QL expression.
    +     +
    Limitations

    +

    The following nest types are currently not supported:

      +
    • Full OUTER NEST
      • +
    • Cross NEST
      • +
    • No mixing of new ANSI NEST syntax with NEST syntax in the same FROM + clause.
      • +
    • The right-hand-side of any nest must be a keyspace. Expressions, subqueries, + or other join combinations cannot be on the right-hand-side of a nest.
      • +
    • A nest can only be executed when appropriate index exists on the inner side + of the ANSI NEST (similar to current NEST support).
      • +
    • Adaptive indexes are not considered when selecting indexes on inner side of + the nest
      • +
    ANSI NEST Example 1: Inner ANSI NEST

    +

    List the airlines, their plane model (equipment), and number of + stops for flights between San Francisco and + Boston.SELECT r.airline, r.equipment, r.stops +FROM `travel-sample` r + NEST `travel-sample` a + ON r.airlineid = META(a).id +WHERE r.sourceairport = "SFO" +AND r.destinationairport = "BOS";Results:[ + { + "airline": "B6", + "equipment": "320", + "stops": 0 + }, + { + "airline": "UA", + "equipment": "752 753 738 739 319 320", + "stops": 0 + }, + { + "airline": "VX", + "equipment": "320", + "stops": 0 + } +]

    +
    +
    Lookup NEST Clause

    (Introduced in Couchbase Server + 4.0)

    Nesting is conceptually the inverse of unnesting. Nesting + performs a join across two keyspaces. But instead of producing a cross-product of + the left and right inputs, a single result is produced for each left input, while + the corresponding right inputs are collected into an array and nested as a single + array-valued field in the result object.

    Syntax

    [ join-type ] NEST from-path [ [ AS ] alias ] on-keys-clause

    +

    + +
    Arguments
    +
    +
    + +
    join-type
    +
    [Optional; default is INNER]
    +
    +
    + +
    INNER
    +
    For each result object produced, both the + left-hand and right-hand source objects must be + non-MISSING and + non-NULL.
    +     +
    +
    + +
    LEFT or LEFT + OUTER
    +
    A left-outer unnest is performed, and at least + one result object is produced for each left source + object.
    +
    For each joined object produced, only the + left-hand source objects must be + non-MISSING and + non-NULL.
    +     +
    +
    +     +
    +
    + +
    from-path
    +
    [Required] Keyspace reference for right-hand side of lookup + nest. For details, see Keyspaces.
    +     +
    +
    + +
    alias (optionally, AS alias)
    +
    [Required] To assign a name for the right-hand side + keyspace. For details, see AS + Keyword.
    +     +
    +
    + +
    on-keys-clause
    +
    [Required] String or expression representing the primary + keys of the documents for the second keyspace.
    +
    The ON KEYS expression produces one or more + document keys for the right-hand side document.
    +
    The ON KEYS expression can produce an array + of document keys.
    +     +
    +
    +     +
    +
    + +
    Return Values
    +
    If the right-hand source object is NULL, MISSING, empty, or a non-array + value, then the result object's right-side value is MISSING + (omitted).
    +
    Nests can be chained with other NEST, JOIN, and UNNEST clauses. By + default, an INNER NEST is performed. This means that for each result + object produced, both the left and right source objects must be + non-missing and non-null. The right-hand side result of NEST is always + an array or MISSING. If there is no matching right source object, then + the right source object is as follows:
    + + + + + + If the ON KEYS expression + evaluates to + Then the right-side value is + + + + + MISSING + MISSING + + + NULL + MISSING + + + an array + an empty array + + + a non-array value + an empty array + + + +
    + + +

    Lookup NEST Example 1: Join two keyspaces producing an output for each + left input.

    Show one set of routes for one airline in the + travel-sample + keyspace.

    SELECT * +FROM `travel-sample` route + INNER NEST `travel-sample` airline + ON KEYS route.airlineid +WHERE route.type = "route" +LIMIT 1;

    Results:

    [ + { + "airline": [ + { + "callsign": "AIRFRANS", + "country": "France", + "iata": "AF", + "icao": "AFR", + "id": 137, + "name": "Air France", + "type": "airline" + } + ], + "route": { + "airline": "AF", + "airlineid": "airline_137", + "destinationairport": "MRS", + "distance": 2881.617376098415, + "equipment": "320", + "id": 10000, + "schedule": [ + { + "day": 0, + "flight": "AF198", + "utc": "10:13:00" + }, + { + "day": 0, + "flight": "AF547", + "utc": "19:14:00" + }, + { + "day": 0, + "flight": "AF943", + "utc": "01:31:00" + }, + { + "day": 1, + "flight": "AF356", + "utc": "12:40:00" + }, + { + "day": 1, + "flight": "AF480", + "utc": "08:58:00" + }, + { + "day": 1, + "flight": "AF250", + "utc": "12:59:00" + }, + { + "day": 1, + "flight": "AF130", + "utc": "04:45:00" + }, + { + "day": 2, + "flight": "AF997", + "utc": "00:31:00" + }, + { + "day": 2, + "flight": "AF223", + "utc": "19:41:00" + }, + { + "day": 2, + "flight": "AF890", + "utc": "15:14:00" + }, + { + "day": 2, + "flight": "AF399", + "utc": "00:30:00" + }, + { + "day": 2, + "flight": "AF328", + "utc": "16:18:00" + }, + { + "day": 3, + "flight": "AF074", + "utc": "23:50:00" + }, + { + "day": 3, + "flight": "AF556", + "utc": "11:33:00" + }, + { + "day": 4, + "flight": "AF064", + "utc": "13:23:00" + }, + { + "day": 4, + "flight": "AF596", + "utc": "12:09:00" + }, + { + "day": 4, + "flight": "AF818", + "utc": "08:02:00" + }, + { + "day": 5, + "flight": "AF967", + "utc": "11:33:00" + }, + { + "day": 5, + "flight": "AF730", + "utc": "19:42:00" + }, + { + "day": 6, + "flight": "AF882", + "utc": "17:07:00" + }, + { + "day": 6, + "flight": "AF485", + "utc": "17:03:00" + }, + { + "day": 6, + "flight": "AF898", + "utc": "10:01:00" + }, + { + "day": 6, + "flight": "AF496", + "utc": "07:00:00" + } + ], + "sourceairport": "TLV", + "stops": 0, + "type": "route" + } + } +]
    +
    + Index NEST Clause +

    (Introduced in Couchbase Server 4.0)

    +

    When Lookup NESTs cannot efficiently nest left-hand side documents with right-to-left + nests and your situation cannot be flipped because your predicate needs to be on the + left-hand side (such as the above Lookup NEST Example #1 where airline documents + have no reference to route documents), then Index NESTs can be used efficiently. + Index NESTs allow you to flip the direction of your nest clause.

    +

    Index NEST Example 1: List four + CREATE INDEX idx_ijoin ON `travel-sample`(airlineid) WHERE type="route"; + +SELECT * +FROM `travel-sample` rte + INNER NEST `travel-sample` aline + ON KEY rte.airlineid + FOR rte +WHERE rte.type = "route" +LIMIT 4;If + you generalize the same query, it looks like the + following:CREATE INDEX on-key-for-index-name rhs-expression (lhs-expression-key); + +SELECT projection-list +FROM lhs-expression + NEST rhs-expression + ON KEY rhs-expression.lhs-expression-key FOR lhs-expression +[ WHERE predicates ] ;There + are three important changes in the index scan syntax example above:

      +
    • CREATE INDEX on the ON KEY + expression route.airlineid to access + route documents using airlineid (which are + produced on the LHS).
      • +
    • The ON KEY route.airlineid FOR airline enables N1QL to use + the index route.airlineid.
      • +
    • Create any optional index, such as route.airline that can + be used on airline (LHS).
      • +
    For index nests, the syntax uses ON KEY (singular) + instead of ON KEYS (plural). This is because Index NESTs' + ON KEY expression must produce a scalar value; whereas + Lookup NESTs' ON KEYS expression can produce either a scalar or + an array value.Syntax

    +

    +

    [ nest-type ] NEST from-path [ [ AS ] alias ] ON KEY on-key-clause FOR for-clauseArguments

    +

    + +
    nest-type
    +
    [Optional; default is LEFT INNER]
    +
    +
    + +
    LEFT or LEFT INNER
    +
    For each nested object produced, both the left-hand and + right-hand source objects must be non-MISSING and + non-NULL.
    +     +
    +
    + +
    LEFT OUTER
    +
    For each nested object produced, only the left-hand source + objects must be non-MISSING and non-NULL.
    +     +
    +
    +     +
    + +
    from-path
    +
    Keyspace reference for right-hand side of an index nest. For details, + see Keyspaces.
    +     +
    + +
    AS alias
    +
    [Optional] To assign another name. For details, see AS Keyword.
    +     +
    + +
    ON KEY rhs-expression.lhs-expression-key
    +
    +
    + +
    rhs-expression
    +
    Keyspace reference for the right-hand side of the index + nest.
    +     +
    +
    + +
    lhs-expression-key
    +
    String or expression representing the attribute in + rhs-expression referencing the document key + for lhs-expression.
    +     +
    +
    +     +
    + +
    FOR lhs-expression
    +
    Keyspace reference for the left-hand side of the index nest.
    +     +
    Index NEST Example 1: ON KEY ... FOR.

    +

    This example nests the airline routes for each airline after creating the following + index. (Note that the index will not match if it contains a WHERE + clause)CREATE INDEX route_airline ON `travel-sample`(airlineid); + +SELECT * +FROM `travel-sample` aline + INNER NEST `travel-sample` rte + ON KEY rte.airlineid + FOR aline +WHERE aline.type = "airline" +LIMIT 1;Results:[ + { + "aline": { + "callsign": "MILE-AIR", + "country": "United States", + "iata": "Q5", + "icao": "MLA", + "id": 10, + "name": "40-Mile Air", + "type": "airline" + }, + "route": [ + { + "airline": "Q5", + "airlineid": "airline_10", + "destinationairport": "HKB", + "distance": 118.20183585107631, + "equipment": "CNA", + "id": 46586, + "schedule": [ + { + "day": 0, + "flight": "Q5188", + "utc": "12:40:00" + }, + { + "day": 0, + "flight": "Q5630", + "utc": "21:53:00" + }, + { + "day": 0, + "flight": "Q5530", + "utc": "07:47:00" + }, + { + "day": 0, + "flight": "Q5132", + "utc": "01:10:00" + }, + { + "day": 0, + "flight": "Q5746", + "utc": "20:11:00" + }, + { + "day": 1, + "flight": "Q5413", + "utc": "08:07:00" + }, + { + "day": 2, + "flight": "Q5263", + "utc": "17:39:00" + }, + { + "day": 2, + "flight": "Q5564", + "utc": "01:55:00" + }, + { + "day": 2, + "flight": "Q5970", + "utc": "00:09:00" + }, + { + "day": 2, + "flight": "Q5295", + "utc": "21:24:00" + }, + { + "day": 2, + "flight": "Q5051", + "utc": "04:41:00" + }, + { + "day": 3, + "flight": "Q5023", + "utc": "00:16:00" + }, + { + "day": 3, + "flight": "Q5554", + "utc": "11:45:00" + }, + { + "day": 3, + "flight": "Q5619", + "utc": "22:22:00" + }, + { + "day": 4, + "flight": "Q5279", + "utc": "23:19:00" + }, + { + "day": 4, + "flight": "Q5652", + "utc": "13:35:00" + }, + { + "day": 4, + "flight": "Q5631", + "utc": "17:53:00" + }, + { + "day": 4, + "flight": "Q5105", + "utc": "21:54:00" + }, + { + "day": 5, + "flight": "Q5559", + "utc": "01:19:00" + }, + { + "day": 5, + "flight": "Q5600", + "utc": "17:36:00" + }, + { + "day": 6, + "flight": "Q5854", + "utc": "22:59:00" + }, + { + "day": 6, + "flight": "Q5217", + "utc": "11:58:00" + }, + { + "day": 6, + "flight": "Q5756", + "utc": "06:32:00" + }, + { + "day": 6, + "flight": "Q5151", + "utc": "15:14:00" + } + ], + "sourceairport": "FAI", + "stops": 0, + "type": "route" + }, + { + "airline": "Q5", + "airlineid": "airline_10", + "destinationairport": "FAI", + "distance": 118.20183585107631, + "equipment": "CNA", + "id": 46587, + "schedule": [ + { + "day": 0, + "flight": "Q5492", + "utc": "17:00:00" + }, + { + "day": 0, + "flight": "Q5357", + "utc": "09:44:00" + }, + { + "day": 0, + "flight": "Q5873", + "utc": "00:01:00" + }, + { + "day": 1, + "flight": "Q5171", + "utc": "00:59:00" + }, + { + "day": 1, + "flight": "Q5047", + "utc": "10:57:00" + }, + { + "day": 1, + "flight": "Q5889", + "utc": "14:51:00" + }, + { + "day": 1, + "flight": "Q5272", + "utc": "18:36:00" + }, + { + "day": 2, + "flight": "Q5673", + "utc": "21:30:00" + }, + { + "day": 3, + "flight": "Q5381", + "utc": "20:01:00" + }, + { + "day": 4, + "flight": "Q5261", + "utc": "18:37:00" + }, + { + "day": 5, + "flight": "Q5755", + "utc": "23:43:00" + }, + { + "day": 5, + "flight": "Q5544", + "utc": "16:04:00" + }, + { + "day": 6, + "flight": "Q5400", + "utc": "10:46:00" + }, + { + "day": 6, + "flight": "Q5963", + "utc": "13:53:00" + }, + { + "day": 6, + "flight": "Q5195", + "utc": "03:03:00" + }, + { + "day": 6, + "flight": "Q5653", + "utc": "22:58:00" + } + ], + "sourceairport": "HKB", + "stops": 0, + "type": "route" + } + ] + } +]

    +
    +
    + Appendix 1 - JOIN Types +

    + + - + + + + - If the ON KEYS expression evaluates - to - The right-side value is + Join + Left-Hand Side (lhs) + Right-Hand Side (rhs) + Syntax + Example - MISSING - MISSING + ANSI + Any field or expr that produces a value that will be matched + on the right-hand side. + Anything that can have a proper index on the join + expression. + lhs-expr

    JOIN + rhs-keyspace

    ON + any join condition

     + SELECT *

    FROM `travel-sample` + r

    JOIN `travel-sample` + a

    ON + r.airlineid = META(a).id

         - NULL - MISSING + Lookup + Must produce a Document Key for the right-hand side. + Must have a Document Key. + lhs-expr

    JOIN + rhs-keyspace

    ON + KEYS

    lhs-expr.foreign_key

     + SELECT *

    FROM `travel-sample` + r

    JOIN `travel-sample` + a

    ON KEYS + r.airlineid

         - an array - an empty array + Index + Must produce a key for the right-hand side's index. + Must have a proper index on the field or expr that maps to + the Document Key of the left-hand side. + lhs-keyspace

    JOIN rhs-keyspace +

    ON KEY + rhs-kspace.idx_key

    FOR + lhs-keyspace

     + SELECT *

    FROM `travel-sample` a +

    JOIN `travel-sample` r +

    ON + KEY + r.airlineid

    FOR + a

     +     + + +
    +

    +
    +
    + Appendix 2 - NEST Types +

    + + + + + + + + + + NEST + Left-Hand Side (lhs) + Right-Hand Side (rhs) + Syntax + Example + + + + + ANSI + Any field or expr that produces a value that will be matched + on the right-hand side. + Anything that can have a proper index on the join + expression. + lhs-expr

    NEST + rhs-keyspace

    ON + any nest condition

     + SELECT *

    FROM `travel-sample` + r

    NEST `travel-sample` + a

    ON + r.airlineid = + META(a).id

     +     + + Lookup + Must produce a Document Key for the right-hand side. + Must have a Document Key. + lhs-expr

    NEST + rhs-keyspace

    ON + KEYS

    lhs-expr.foreign_key

     + SELECT *

    FROM `travel-sample` + r

    NEST `travel-sample` a +

    ON KEYS + r.airlineid

    WHERE + r.type="route"

    LIMIT + 4;

         - a non-array value - an empty array + Index + Must produce a key for the right-hand side index. + Must have a proper index on the field or expr that maps to + the Document Key of the left-hand side. + lhs-keyspace

    NEST rhs-keyspace +

    ON KEY + rhs-kspace.idx_key

    FOR + lhs-keyspace

     + SELECT *

    FROM `travel-sample` + a

    NEST `travel-sample` + r

    ON KEY + r.airlineid +

    FOR + a

    WHERE + a.type="airline"

    LIMIT + 4;

         -

    - -

    If LEFT or LEFT OUTER is specified, then a left outer nest - is performed. One result object is produced for each left source object.

    - -

    Example

    -

    This example shows the NEST clause using invoice and - invoice_item objects.

    -

    Recall our invoice objects:

    - { - "customer_key": ..., - "invoice_date": ..., - "invoice_item_keys": [ ... ], - "total": ... - } -

    And our invoice_item objects:

    - { - "invoice_key": ..., - "product_key": ..., - "unit_price": ..., - "quantity": ..., - "item_subtotal": ... -} -

    If the FROM clause is:

    - FROM invoice inv NEST invoice_item items ON KEYS inv.invoice_item_keys -

    Then the results would be:

    - { - "invoice" : { - "customer_key": ..., - "invoice_date": ..., - "invoice_item_keys": [ ... ], - "total": ... - }, - "items" : [ - { - "invoice_key": ..., - "product_key": ..., - "unit_price": ..., - "quantity": ..., - "item_subtotal": ... - }, - { - "invoice_key": ..., - "product_key": ..., - "unit_price": ..., - "quantity": ..., - "item_subtotal": ... - } - ] - }, - { - "invoice" : { - "customer_key": ..., - "invoice_date": ..., - "invoice_item_keys": [ ... ], - "total": ... - }, - "items" : [ - { - "invoice_key": ..., - "product_key": ..., - "unit_price": ..., - "quantity": ..., - "item_subtotal": ... - }, - { - "invoice_key": ..., - "product_key": ..., - "unit_price": ..., - "quantity": ..., - "item_subtotal": ... - } - ] - }, - ... -
    -
    - Arrays -

    If an array occurs along a path, you can use array subscripts to select one element.

    -

    In the following statement, the entire address array is selected for each customer.

    - SELECT a FROM customer.address a; - [ - { - "a": [ - { "street" : "101 Main St.", "zip" : "94040" }, - { "street" : "300 Broadway", "zip" : "10011" } - ] - }, - { - "a": [ - { "street" : "3500 Wilshire Blvd.", "zip" : "90210" }, - { "street" : "4120 Alamo Dr.", "zip" : "75019" } - ] - } -] -

    The following example uses array subscripts to select the first element of the - address array for each customer.

    - SELECT * FROM customer.address[0] - [ - { "street" : "101 Main St.", "zip" : "94040" }, - { "street" : "3500 Wilshire Blvd.", "zip" : "90210" } -] + +

    -
    AS Keyword - -

    Like SQL, N1QL allows renaming fields using the AS keyword. However, N1QL - also allows reshaping the data, which has no analog in SQL. To do this, you embed - the attributes of the statement in the desired result object shape.

    -
    -
    Aliases -

    Aliases in the FROM clause create new names that can be referred to - anywhere in the query. When an alias conflicts with a keyspace or field name in the - same scope, the identifier always refers to the alias. This allows for consistent - behavior in scenarios where an identifier only conflicts in some documents. For more - information on aliases, see .

    -
    - - + + diff --git a/content/n1ql/n1ql-language-reference/groupby-aggregate-performance.dita b/content/n1ql/n1ql-language-reference/groupby-aggregate-performance.dita new file mode 100644 index 0000000000..2e94069e3d --- /dev/null +++ b/content/n1ql/n1ql-language-reference/groupby-aggregate-performance.dita @@ -0,0 +1,3037 @@ + + + + Group By and Aggregate Performance + N1QL Pushdowns optimize the performance of N1QL queries by supporting GROUP BY and + Aggregate expressions. + +

    (Introduced in Couchbase Server 5.5 - Enterprise Edition)

    +
    + Overview +

    This features improves performance of N1QL queries with aggregations and GROUP BY + execution.

    +

    After the optimizer selects an index for a query block, it attempts the two + optimizations below:

      +
    • Pagination optimization, by pushing the OFFSET and LIMIT parameters to the + index scan.
      • +
    • Grouping and aggregation pushdown to the indexer (introduced in Couchbase + 5.5).
      • +
    Prior to Couchbase 5.5, even when a query with aggregate and/or GROUP BY is + covered by an index, the query fetched all relevant data from the indexer and group + the data within the query engine. With this enhancement, the query intelligently + requests the indexer to perform grouping and aggregation in addition to range scan. + The Indexer has been enhanced to perform grouping, COUNT(), SUM(), MIN(), MAX(), + AVG(), and related operations.

    +

    This requires no changes to the user query, but a good index design to cover the + query and order the keys is required. Not every query will benefit from this + optimization, and not every index can accelerate every grouping and aggregation. + Understanding the right patterns will help you to design your indexes and queries. + Aggregate Pushdown to the global secondary index is supported on both storage + engines: Standard GSI and Memory Optimized GSI (MOI).

    +

    This reduction step of performing the GROUP BY and Aggregation on the indexer reduces + the amount of data transfer and disk I/O, resulting in:

      +
    • Improved query response time
      • +
    • Improved resource utilization
      • +
    • Low latency
      • +
    • High scalability
      • +
    • Low TCO
      • +
    For example, let's compare the previous vs. current performance of using GROUP + BY and examine the EXPLAIN plan of the following query that is defined in the + Couchbase travel-sample + index:CREATE INDEX `def_type` ON `travel-sample`(`type`) Consider + the + query:SELECT type, COUNT(type) +FROM `travel-sample` +WHERE type IS NOT MISSING +GROUP BY type;Before + Couchbase version 5.5, this query engine fetched relevant data from the indexer and + group the data within the query engine and then aggregate. This simple query takes + about 250 ms.

    +

    +

    Now, in Couchbase version 5.5, this query use the same def_type + index, but executes in under 70 ms. In the explain below, you can see fewer steps + and the lack of the grouping step after the index scan because the index scan step + does the grouping and aggregation as well.

    +

    +

    As the data and query complexity grows, the performance benefit (both latency and + throughput) will grow as well.

    +

    For example, consider the + query:SELECT type, COUNT(type) +FROM `travel-sample` +WHERE type IS NOT MISSING +GROUP BY type;

    +

    The text explain plan shows the accelerated aggregation details. For details, see the + Query Plan Fields + section.... + "index_group_aggs": { + "aggregates": [ + { + "aggregate": "COUNT", + "expr": "1", + "id": 2, + "keypos": -1 + } + ], + "depends": [ + 0 + ], + "group": [ + { + "depends": [ + 0 + ], + "expr": "cover ((`travel-sample`.`type`))", + "id": 0, + "keypos": 0 + } + ] + }, + "index_id": "c5bbb792c69c1704", + "index_projection": { + "entry_keys": [ + 0, + 2 + ] + }, +...Here’s + is how this query executes when the indexer executes the GROUP BY and aggregation. + The query engine does not fetch any data from the data service (KV service), as + shown below:

    +

    +

    For your reference, this is how the same query executed before 5.5.

    +

    +
    +
    Examples for Indexer GROUP BY and + Aggregation

    Example A: Let’s Consider a composite index to explore some + scenarios:CREATE INDEX idx_a ON `travel-sample` (geo.alt, geo.lat, geo.lon, id) WHERE type = "airport"Let’s + consider sample queries that can benefit from this optimization and the queries that + cannot.

    Positive Case examples of queries that use indexing grouping and + aggregation:

      +
    • SELECT COUNT(*) FROM `travel-sample` WHERE geo.alt > 10 AND + type="airport";
      • +
    • SELECT COUNT(geo.alt) FROM `travel-sample` WHERE geo.alt BETWEEN 10 AND 30 AND + type = "airport";
      • +
    • SELECT COUNT(geo.lat) FROM `travel-sample` WHERE geo.alt BETWEEN 10 AND 30 AND + geo.lat = 40 AND type = "airport";
      • +
    • SELECT geo.alt, AVG(id), SUM(id), COUNT(geo.alt), MIN (geo.lon), + MAX(ABS(geo.lon)) FROM `travel-sample` WHERE geo.alt > 100 AND type = "airport" GROUP + BY geo.alt;
      • +
    • SELECT lat_count, SUM(id) FROM `travel-sample` WHERE geo.alt > 100 AND type = + "airport" GROUP BY geo.alt LETTING lat_count = COUNT(geo.lat) HAVING lat_count > + 1;
      • +
    • SELECT AVG(DISTINCT geo.lat) FROM `travel-sample` WHERE geo.alt > 100 AND type + = "airport" GROUP BY geo.alt;
      • +

    Negative Case examples:

      +
    • SELECT COUNT(*) FROM `travel-sample` WHERE geo.lat > 20 AND type = + "airport";
        +
      • This query has no predicate on the leading key geo.alt. The index + idx_a cannot be used.
        • +
      • +
    • SELECT COUNT(*) FROM `travel-sample`;
        +
      • This query has no predicate at all.
        • +
      • +
    • SELECT COUNT(v1) FROM `travel-sample` LET v1 = ROUND(geo.lat) WHERE geo.lat > + 10 AND type = "airport";
        +
      • The aggregate depends on LET variable.
        • +
      • +
    • SELECT ARRAY_AGG(geo.alt) FROM `travel-sample` WHERE geo.alt > 10 AND type = + "airport";
        +
      • ARRAY_AGG is not supported.
        • +
      • +

    Positive query examples with GROUP BY on leading index + keys

    Example B: Consider the following + index:CREATE INDEX idx_b ON `travel-sample`(geo.alt, geo.lat, geo.lon, id)In + the following query, the GROUP BY keys (geo.alt, geo.lat) are the leading + keys of the index, so the index is naturally ordered and grouped by the order of the index + key definition. Therefore, the query below is suitable for indexer to handle grouping and + aggregation.SELECT geo.alt, geo.lat, SUM(geo.lon), AVG(id), COUNT(DISTINCT geo.lon) +FROM `travel-sample` +WHERE geo.alt BETWEEN 10 AND 30 +AND type = "airport" +GROUP BY geo.alt, geo.lat +HAVING SUM(geo.lon) > 1000;Here's + the executed query plan showing that index scan handled grouping and aggregation:

    Positive query examples with + GROUP BY on non-leading index keys

    Example C: Consider the following + index and + query:CREATE INDEX idx_c ON `travel-sample`(geo.alt, geo.lat, geo.lon, id) + WHERE type = "airport" + +SELECT geo.lat, id, SUM(geo.lon) +FROM `travel-sample` +WHERE geo.alt BETWEEN 10 AND 30 +AND type = "airport" +GROUP BY geo.lat, id +HAVING SUM(geo.lon) > 1000;The + following is a bottom-up rendering of the execution plan for easier viewing. In this case, + the indexer sends partial group aggregation, which the query merges to create the final + group and aggregation. In this scenario (when the grouping is on non-leading keys), any + query with aggregation and DISTINCT modifier cannot be accelerated by the indexer, such as + COUNT(DISTINCT id).

    Positive query examples on + array indexes with GROUP BY on leading index keys

    Example D: Consider the + following index and + query:CREATE INDEX idx_d ON `travel-sample` (geo.lat, geo.lon, DISTINCT public_likes, id) WHERE type = "hotel" + +SELECT geo.lat, geo.lon, SUM(id), AVG(id) +FROM `travel-sample` +WHERE geo.lat BETWEEN 10 AND 30 + AND geo.lon > 50 + AND type = "hotel" + AND ANY v IN public_likes SATISFIES v = “%a%” END +GROUP BY geo.lat, geo.lon +HAVING SUM(id) > 100;In + this case, the predicates are on the leading keys up to and including the array key. + Therefore, indexer can efficiently do the grouping as seen by the optimal plan below. It’s + important to note the array index key is created with a DISTINCT modifier + (not the ALL modifier) to get this optimization and that the + SATISFIES clause in the ANY predicate must be that of + equality (that is, v = “%a%”).

    Example D2: On the other hand, if there’s a + predicate missing on geo.lon--which is prior to the array key--while using + the same idx_d index as above, then the grouping is done by the old + method:SELECT geo.lat, geo.lon, SUM(id), AVG(id) +FROM `travel-sample` +WHERE geo.lat BETWEEN 10 AND 30 + AND type = "hotel" + AND ANY v IN public_likes SATISFIES v = “%a%” END +GROUP BY geo.lat, geo.lon +HAVING SUM(id) > 100;

    Example E: Consider the + index and + query:CREATE INDEX idx_e ON `travel-sample` (ALL public_likes, geo.lat, geo.lon, id) WHERE type = "hotel" + +SELECT un, t.geo.lat, COUNT(un), AVG(t.geo.lat) +FROM `travel-sample` AS t + UNNEST t.public_likes AS un +WHERE un > "J" +AND t.type = "hotel" +GROUP BY un, t.geo.lat;In + this case, the UNNEST operation can use the index because the leading + ALL array key is the array being unwound. Note, the unwound operation + repeats the parent document (travel-sample) and the + t.geo.lat reference would have duplicates compared to the original + travel-sample documents.

    +
    + Query Qualification and Pushdown +

    Not every GROUP BY and aggregate query can be handled by the indexer. Following are some + simple rules that will help you to write the proper queries and design the required indexes + to get the most of this feature.

    +

    The following are necessary in order for an indexer to execute GROUP BY and aggregates:

      +
    • All the query predicates are able to convert into ranges and able to push to + indexer.
      • +
    • The whole query must be covered by an index.
        +
      • For a query to be covered by an index, every attribute referenced in the query + should be in one index.
        • +
      • Query should not have operations such as joins, subquery, or derived table + queries.
        • +
      • +
    • GROUP BY keys and Aggregate expressions must be one of the following:
        +
      • Index keys or document key
        • +
      • An expression based on index keys or document key
        • +
      • +
    • GROUP BY and aggregate expressions must be simple.
      • +

    +
    +
    + Scenarios for Group By and Aggregation +

    Like any feature in a query language, there are subtle variations between each query + and index that affects this optimization. We use the travel-sample + dataset to illustrate both positive and negative use cases.

    +

    The following table lists the scenarios and requirements for queries to request the + indexer to do the grouping and acceleration. When the requirements are unmet, the + query will fetch the relevant data and then do the grouping and acceleration as + usual. No application changes are necessary. The query plan generated reflects this + decision.

    +

    +

    + +
    GROUP BY Scenarios:
    +
    1. GROUP BY on leading keys
    +
    2. GROUP BY on non-leading keys
    +
    3. GROUP BY keys in different CREATE INDEX + order
    +
    4. GROUP BY on expression
    +
    5. Heterogeneous data types for GROUP BY key
    +
    6. GROUP BY META().id Primary Index
    +
    7. LIMIT with GROUP BY on leading keys
    +
    8. OFFSET with GROUP BY on leading keys
    +     +
    +
    + +
    Aggregate Scenarios:
    +
    9. Aggregate without GROUP BY
    +
    10. Expression in aggregate function
    +
    11. SUM, COUNT, MIN, MAX, or AVG Aggregate + function
    +
    12. DISTINCT aggregates
    +
    13. HAVING with an aggregate function inside
    +
    14. LETTING with an aggregate function inside
    +     +
    + +

    +

    1. GROUP BY on + leading keys

    +

    One of the common cases is to have both predicates and GROUP BY on leading keys of + the index. First create the index so that the query is covered by the index. You can + then think about the order of the keys.

    +

    The query requires a predicate on leading keys to consider an index. The simplest + predicate is IS NOT MISSING. + CREATE INDEX idx_expr ON Keyspace_ref (a, b, c); + +SELECT a, b, Aggregate_Function(c) /* MIN(c), MAX(c), COUNT(c), or SUM(c) */ +FROM Keyspace_ref +WHERE a IS NOT MISSING /* 1st index field must be in a WHERE clause */ +GROUP BY a, b;Example + 1: List the cities with the landmarks with the highest latitude.

    +

    Use the MAX() aggregate to find the highest landmark latitude in + each state, group the results by country and + state, and then sort in reverse order by the highest latitudes per + state.CREATE INDEX idx1 ON `travel-sample`(country, state, geo.lat) +WHERE type="landmark"; + +SELECT country, state, MAX(ROUND(geo.lat)) AS Max_Latitude +FROM `travel-sample` +WHERE country IS NOT MISSING +AND type = "landmark" +GROUP BY country, state +ORDER BY Max_Latitude DESC;In + this query, we need to give the predicate country IS NOT MISSING + (or any WHERE clause) to ensure this index is selected for the query. + Without a matching predicate, the query will use the primary index.

    +

    Results:[ + { + "Max_Latitude": 60, + "country": "United Kingdom", + "state": null + }, + { + "Max_Latitude": 51, + "country": "United Kingdom", + "state": "England" + }, + { + "Max_Latitude": 50, + "country": "France", + "state": "Picardie" + }, +...The + Example 1 EXPLAIN Plan shows that GROUP BY is executed by the + indexer and is detailed in the GROUP BY + Query Plan + table:{ + "plan": { + "#operator": "Sequence", + "~children": [ + { + "#operator": "Sequence", + "~children": [ + { + "#operator": "IndexScan3", + "covers": [ + "cover ((`travel-sample`.`country`))", + "cover ((`travel-sample`.`state`))", + "cover (((`travel-sample`.`geo`).`lat`))", + "cover ((meta(`travel-sample`).`id`))", + "cover (count(cover ((`travel-sample`.`country`))))", + "cover (min(round(cover (((`travel-sample`.`geo`).`lat`)))))" + ], + "filter_covers": { + "cover ((`travel-sample`.`type`))": "landmark" + }, + "index": "idx1", + "index_group_aggs": { + "aggregates": [ + { + "aggregate": "MAX", + "depends": [ + 2 + ], + "expr": "round(cover (((`travel-sample`.`geo`).`lat`)))", + "id": 4, + "keypos": -1 + } + ], + "depends": [ + 0, + 1, + 2 + ], + "group": [ + { + "depends": [ + 0 + ], + "expr": "cover ((`travel-sample`.`country`))", + "id": 0, + "keypos": 0 + }, + { + "depends": [ + 1 + ], + "expr": "cover ((`travel-sample`.`state`))", + "id": 1, + "keypos": 1 + } + ] + }, +...

    +

    2. GROUP BY on + non-leading keys

    +

    When using GROUP BY on a non-leading key:

      +
    • The indexer will return pre-aggregated results.
      • +
    • Results can have duplicate or out-of-order groups. The N1QL indexer will do + 2nd level of aggregation and compute the final result.
      • +
    • The N1QL indexer can pushdown only if the leading key has a predicate.
      • +
    To use Aggregate Pushdown, use the following syntax of the index and query + statements:CREATE INDEX idx_expr ON Keyspace_ref (a, b, c);Syntax + A:SELECT Aggregate_Function(a), b, Aggregate_Function(c) +FROM Keyspace_ref +WHERE a IS NOT MISSING +GROUP BY b;Syntax + B:SELECT Aggregate_Function(a), Aggregate_Function(b), c +FROM Keyspace_ref +WHERE a IS NOT MISSING +GROUP BY c;Example + 2 (A): List the states with their total number of landmarks and the lowest + latitude of any landmark.

    +

    Use the COUNT() operator to find the total number of landmarks and + use the MIN() operator to find the lowest landmark latitude in each + state, group the results by state, and then sort in order by the + lowest latitudes per + state.CREATE INDEX idx2 ON `travel-sample`(country, state, ROUND(geo.lat)) +WHERE type="landmark"; + +SELECT COUNT(country) AS Total_landmarks, state, MIN(ROUND(geo.lat)) AS Min_Latitude +FROM `travel-sample` +WHERE country IN ["France", "United States", "United Kingdom"] +AND type = "landmark" +GROUP BY state +ORDER BY Min_Latitude;Explain + Plan:

    +

    { + "plan": { + "#operator": "Sequence", + "~children": [ + { + "#operator": "Sequence", + "~children": [ + { + "#operator": "IndexScan3", + "covers": [ + "cover ((`travel-sample`.`country`))", + "cover ((`travel-sample`.`state`))", + "cover (((`travel-sample`.`geo`).`lat`))", + "cover ((meta(`travel-sample`).`id`))", + "cover (count(cover ((`travel-sample`.`country`))))", + "cover (min(round(cover (((`travel-sample`.`geo`).`lat`)))))" + ], + "filter_covers": { + "cover ((`travel-sample`.`type`))": "landmark" + }, + "index": "idx2", + "index_group_aggs": { + "aggregates": [ + { + "aggregate": "COUNT", + "depends": [ + 0 + ], + "expr": "cover ((`travel-sample`.`country`))", + "id": 4, + "keypos": 0 + }, + { + "aggregate": "MIN", + "depends": [ + 2 + ], + "expr": "round(cover (((`travel-sample`.`geo`).`lat`)))", + "id": 5, + "keypos": -1 + } + ], + "depends": [ + 0, + 1, + 2 + ], + "group": [ + { + "depends": [ + 1 + ], + "expr": "cover ((`travel-sample`.`state`))", + "id": 1, + "keypos": 1 + } + ], + "partial": true + }, +...The + "partial": true line means it was + pre-aggregated.Results:[ + { + "Min_Latitude": 33, + "Total_landmarks": 1900, + "state": "California" + }, + { + "Min_Latitude": 41, + "Total_landmarks": 8, + "state": "Corse" + }, + { + "Min_Latitude": 43, + "Total_landmarks": 6, + "state": "Languedoc-Roussillon" + }, +...Example + 2 (B): List the number of landmarks by latitude and the state it's in.

    +

    Use COUNT(country) for the total number of landmarks at each + latitude. At a particular latitude, the state will be the same; but + an aggregate function on it is needed, so MIN() or + MAX() is used to return the original + value.SELECT COUNT(country) Num_Landmarks, MIN(state) State_Name, ROUND(geo.lat) Latitude +FROM `travel-sample` +WHERE country IS NOT MISSING +AND type = "landmark" +GROUP BY ROUND(geo.lat) +ORDER BY ROUND(geo.lat);Results:[ + { + "Latitude": 33, + "Num_Landmarks": 227, + "State_Name": "California" + }, + { + "Latitude": 34, + "Num_Landmarks": 608, + "State_Name": "California" + }, + { + "Latitude": 35, + "Num_Landmarks": 27, + "State_Name": "California" + }, +...3. GROUP BY keys in + different CREATE INDEX order

    +

    When using GROUP BY on keys in a different order than they appear in the CREATE INDEX + statement, use the following + syntax:CREATE INDEX idx_expr ON Keyspace_ref(a, b, c); + +SELECT Aggregate_Function(c) +FROM Keyspace_ref +WHERE a IS NOT MISSING +GROUP BY b, a;Example + 3: Like Example 1 with the GROUP BY fields swapped, list the landmarks with + the lowest longitude.

    +

    Use the MIN() operator to find the lowest landmark longitude in each + city, group the results by activity and city, and + then sort in reverse order by the lowest longitudes per + activity.CREATE INDEX idx3 ON `travel-sample`(activity, city, geo.lon) +WHERE type="landmark"; + +SELECT activity, city, MIN(ROUND(geo.lon)) AS Max_Longitude +FROM `travel-sample` +WHERE country IS NOT MISSING +AND type = "landmark" +GROUP BY activity, city +ORDER BY Min_Longitude;Results:[ + { + "Min_Longitude": -124, + "activity": "buy", + "city": "Eureka" + }, + { + "Min_Longitude": -123, + "activity": "drink", + "city": "Glen Ellen" + }, + { + "Min_Longitude": -123, + "activity": "do", + "city": "Santa Rosa" + }, + { + "Min_Longitude": -123, + "activity": "eat", + "city": "Moss Beach" + }, +...4. GROUP BY on + expression

    +

    When grouping on an expression or operation, the indexer will return pre-aggregated + results whenever the GROUP BY and leading index keys are not an exact match.

    +

    To use Aggregate Pushdown and avoid pre-aggregated results, use one of the two + following syntaxes of the index and query statements:

    +

    Syntax A: Field with an expression (GROUP BY and Index keys + match)CREATE INDEX idx_expr ON Keyspace_ref(a+b, b, c); + +SELECT Aggregate_Function(c) +FROM Keyspace_ref +WHERE a IS NOT MISSING +GROUP BY a+b;Syntax + B: Operation on a field (GROUP BY and Index keys + match)CREATE INDEX idx_operation ON Keyspace_ref (LOWER(a), b, c); + +SELECT Aggregate_Function(c) +FROM Keyspace_ref +WHERE a IS NOT MISSING +GROUP BY LOWER(a);For + comparison, the below index and query combination will yield pre-aggregated + results.

    +

    Pre-aggregated Syntax: The GROUP BY and Index keys don't + match.CREATE INDEX idx_operation ON Keyspace_ref (a, b, c); + +SELECT Aggregate_Function(c) +FROM Keyspace_ref +WHERE a IS NOT MISSING +GROUP BY UPPER(a);Example + 4 (A): A field with an expression.

    +

    Let's say the distance of a flight feels like "nothing" when it's direct, but feels + like the true distance when there is one layover. Then we can list and group by + flight distances by calculating the distance multiplied by the stops it + makes.CREATE INDEX idx4_expr +ON `travel-sample`(ROUND(distance*stops), ROUND(distance), sourceairport) +WHERE type="route"; + +SELECT ROUND(distance*stops) AS Distance_Feels_Like, + MAX(ROUND(distance)) AS Distance_True, + COUNT(sourceairport) Number_of_Airports +FROM `travel-sample` +WHERE ROUND(distance*stops) IS NOT MISSING +AND type = "route" +GROUP BY ROUND(distance*stops);Query + Plan:

    +

    ... + "index_group_aggs": { + "aggregates": [ + { + "aggregate": "COUNT", + "depends": [ + 2 + ], + "expr": "cover ((`travel-sample`.`sourceairport`))", + "id": 4, + "keypos": 2 + }, + { + "aggregate": "MAX", + "depends": [ + 1 + ], + "expr": "cover (round((`travel-sample`.`distance`)))", + "id": 5, + "keypos": 1 + } + ], + "depends": [ + 0, + 1, + 2 + ], + "group": [ + { + "depends": [ + 0 + ], + "expr": "cover (round(((`travel-sample`.`distance`) * (`travel-sample`.`stops`))))", + "id": 0, + "keypos": 0 + } + ] +...Results:[ + { + "Distance_Feels_Like": 1055, + "Distance_True": 1055, + "Number_of_Airports": 1 + }, + { + "Distance_Feels_Like": 1806, + "Distance_True": 1806, + "Number_of_Airports": 2 + }, + { + "Distance_Feels_Like": 0, + "Distance_True": 13808, + "Number_of_Airports": 24018 + }, +...Example + 4 (B): An operation on a field.

    +

    Let's say the distance of a flight feels like "nothing" when it's direct, but feels + like the true distance when there is one layover. Then we can list and group by the + uppercase of the airport codes and listing the flight distances by calculating the + distance multiplied by the stops it makes along with the total + distance.CREATE INDEX idx4_oper +ON `travel-sample`(sourceairport, ROUND(distance*stops), distance) +WHERE type="route"; + +SELECT UPPER(sourceairport) AS Airport_Code, + MIN(ROUND(distance*stops)) AS Distance_Feels_Like, + SUM(ROUND(distance)) AS Total_Distance +FROM `travel-sample` +WHERE sourceairport IS NOT MISSING +AND type = "route" +GROUP BY UPPER(sourceairport);Results:[ + { + "Airport_Code": "ESU", + "Distance_Feels_Like": 0, + "Total_Distance": 6223 + }, + { + "Airport_Code": "QSF", + "Distance_Feels_Like": 0, + "Total_Distance": 3285 + }, + { + "Airport_Code": "LHW", + "Distance_Feels_Like": 0, + "Total_Distance": 13837 + }, +...5. Heterogeneous + data types for GROUP BY key

    +

    When a field has a mix of data types for the GROUP BY key:

      +
    • NULLS and MISSING are two separate groups.
      • +
    Example 5:

    +

    To see a separate grouping of MISSING and NULL, we need + to GROUP BY a field we know exists in one document but not in another + document while both documents have another field in common. For example, create 3 such + documents:INSERT INTO `travel-sample` VALUES("01",{"type":1, "email":"abc","xx":3}); + +INSERT INTO `travel-sample` VALUES("01",{"type":1, "email":"abc","xx":null}); + +INSERT INTO `travel-sample` VALUES("02",{"type":1, "email":"abcd"});Then + run the following + query:SELECT type, xx, MIN(email) AS Min_Email +FROM `travel-sample` +WHERE type IS NOT NULL +GROUP BY type, xx;Results:[ + { + "Min_Email": "abc", + "type": 1, + "xx": 3 + }, + { + "Min_Email": "abc", + "type": 1, + "xx": null + }, + { + "Min_Email": "abcd", + "type": 1 <-- is a separate result since field "xx" is MISSING + }, + { + "Min_Email": null, + "type": "airline" + }, +...6. GROUP BY META().ID Primary + Index

    +

    If there is no filter, then pushdown is supported for an expression on the Document + ID META().id in the GROUP BY clause.

    +

    To use Aggregate Pushdown, use the following example of the index and query + statement:CREATE PRIMARY INDEX idx_expr ON named_keyspace_ref; + +SELECT COUNT(1) +FROM named_keyspace_ref +GROUP BY SUBSTR(META().id, 0, 10);If + there is a filter on the Document ID, then the primary index can be used as a secondary + scan.Example 6: List the number of countries that are in each decile of the + META().id + field.CREATE PRIMARY INDEX idx6 ON `travel-sample`; + +SELECT COUNT(1) AS Cnt, SUBSTR(META().id,0,9) AS Meta_Group +FROM `travel-sample` +GROUP BY SUBSTR(META().id,0,9);Results:[ + { + "Meta_Group": "airport_9", + "Number_of_Country": 121 + }, + { + "Meta_Group": "airport_1", + "Number_of_Country": 187 + }, + { + "Meta_Group": "airport_3", + "Number_of_Country": 482 + }, +...7. LIMIT with GROUP BY on + leading keys

    +

    To use Aggregate Pushdown when there is a LIMIT clause and a GROUP BY clause on one + or more leading keys, use the following example of the index and query + statement:CREATE INDEX idx_expr ON named_keyspace_ref (k0, k1); + +SELECT k0, COUNT(k1) +FROM named_keyspace_ref +WHERE k0 IS NOT MISSING +GROUP BY k0 +LIMIT n;Example + 7: LIMIT with GROUP BY on the leading + key.CREATE INDEX idx7 ON `travel-sample` (city, name) +WHERE type = "landmark"; + +SELECT city AS City, COUNT(DISTINCT name) AS Landmark_Count +FROM `travel-sample` +WHERE city IS NOT MISSING +AND type = "landmark" +GROUP BY city +LIMIT 4;Explain + Plan:{ + "plan": { + "#operator": "Sequence", + "~children": [ + { + "#operator": "Sequence", + "~children": [ + { + "#operator": "IndexScan3", + "covers": [ + "cover ((`travel-sample`.`city`))", + "cover ((`travel-sample`.`name`))", + "cover ((meta(`travel-sample`).`id`))", + "cover (count(distinct cover ((`travel-sample`.`name`))))" + ], + "filter_covers": { + "cover ((`travel-sample`.`type`))": "landmark" + }, + "index": "idx7", + "index_group_aggs": { + "aggregates": [ + { + "aggregate": "COUNT", + "depends": [ + 1 + ], + "distinct": true, + "expr": "cover ((`travel-sample`.`name`))", + "id": 3, + "keypos": 1 + } + ], + "depends": [ + 0, + 1 + ], + "group": [ + { + "depends": [ + 0 + ], + "expr": "cover ((`travel-sample`.`city`))", + "id": 0, + "keypos": 0 + } + ] + }, + "index_id": "7852b5e2c07281f3", + "index_projection": { + "entry_keys": [ + 0, + 3 + ] + }, + "keyspace": "travel-sample", + "limit": "4", + "namespace": "default", +...The + limit is pushed to the indexer because the GROUP BY key matched with + the leading index + key.Results:[ + { + "City": null, + "Landmark_Count": 15 + }, + { + "City": "Abbeville", + "Landmark_Count": 1 + }, + { + "City": "Abbots Langley", + "Landmark_Count": 19 + }, + { + "City": "Aberdeenshire", + "Landmark_Count": 6 + } +]8. OFFSET with GROUP BY on + leading keys

    +

    To use Aggregate Pushdown when there is an OFFSET clause and a GROUP BY clause on one + or more leading keys, use the following example of the index and query + statement.CREATE INDEX idx_expr ON named_keyspace_ref (k0, k1); + +SELECT k0, COUNT(k1) +FROM named_keyspace_ref +WHERE k0 IS NOT MISSING +GROUP BY k0 +OFFSET n;Example + 8: OFFSET with GROUP BY on a leading + key.CREATE INDEX idx8 ON `travel-sample` (city, name) +WHERE type = "landmark"; + +SELECT city AS City, COUNT(DISTINCT name) AS Landmark_Count +FROM `travel-sample` +WHERE city IS NOT MISSING +AND type = "landmark" +GROUP BY city +OFFSET 4;Explain + Plan:{ + "plan": { + "#operator": "Sequence", + "~children": [ + { + "#operator": "Sequence", + "~children": [ + { + "#operator": "IndexScan3", + "covers": [ + "cover ((`travel-sample`.`city`))", + "cover ((`travel-sample`.`name`))", + "cover ((meta(`travel-sample`).`id`))", + "cover (count(distinct cover ((`travel-sample`.`name`))))" + ], + "filter_covers": { + "cover ((`travel-sample`.`type`))": "landmark" + }, + "index": "idx7", + "index_group_aggs": { + "aggregates": [ + { + "aggregate": "COUNT", + "depends": [ + 1 + ], + "distinct": true, + "expr": "cover ((`travel-sample`.`name`))", + "id": 3, + "keypos": 1 + } + ], + "depends": [ + 0, + 1 + ], + "group": [ + { + "depends": [ + 0 + ], + "expr": "cover ((`travel-sample`.`city`))", + "id": 0, + "keypos": 0 + } + ] + }, + "index_id": "7852b5e2c07281f3", + "index_projection": { + "entry_keys": [ + 0, + 3 + ] + }, + "keyspace": "travel-sample", + "namespace": "default", + "offset": "4", + "spans": [ +...The + offset is pushed to the indexer because the GROUP BY key matched with + the leading index + key.Results:[ + { + "City": "Aberdour", + "Landmark_Count": 4 + }, + { + "City": "Aberdulais", + "Landmark_Count": 1 + }, + { + "City": "Abereiddy", + "Landmark_Count": 1 + }, + { + "City": "Aberfeldy", + "Landmark_Count": 2 + }, +...9. Aggregate without GROUP BY + key

    +

    This is a case of aggregation over a range without groups. If the index can be used + for computing the aggregate, the indexer will return a single aggregate value. To + use Aggregate Pushdown, use the following syntax of index and + queries:CREATE INDEX idx_expr ON named_keyspace_ref (a, b, c); + +Q1: SELECT Aggregate_Function(c) + FROM named_keyspace_ref + WHERE a IS NOT MISSING; + +Q2: SELECT SUM(a) + FROM named_keyspace_ref + WHERE a IS NOT MISSING; + +Q3: SELECT SUM(a), COUNT(a), MIN(a) + FROM named_keyspace_ref + WHERE a IS NOT MISSING; + +Q4: SELECT SUM(a), COUNT(b), MIN(c) + FROM named_keyspace_ref + WHERE a IS NOT MISSING;Example + 9 (Q1): Multiple Aggregate without GROUP BY + key.CREATE INDEX idx9 +ON `travel-sample`(ROUND(distance), stops, sourceairport) +WHERE type = "airport"; + +Q1: SELECT SUM(ROUND(distance)) AS Total_Distance, + SUM(stops) AS Total_Stops, + COUNT(sourceairport) AS Total_Airports + FROM `travel-sample` + WHERE distance IS NOT MISSING + AND type = "airport";Results:[ + { + "Total_Airports": 24024, + "Total_Distance": 53538071, + "Total_Stops": 6 + } +]Example + 9 (Q2): Aggregate without GROUP BY + key.Q2: SELECT SUM(ROUND(distance)) AS Total_Distance + FROM `travel-sample`;Results:[ + { + "Total_Distance": 53538071 + } +]Example + 9 (Q3): Multiple Aggregate without GROUP BY + key.Q3: SELECT SUM(ROUND(distance)) AS Total_Distance, + COUNT(ROUND(distance)) AS Count_of_Distance, + MIN(ROUND(distance)) AS Min_of_Distance + FROM `travel-sample` + WHERE distance IS NOT MISSING;Results:[ + { + "Count_of_Distance": 24024, + "Min_of_Distance": 3, + "Total_Distance": 53538071 + } +]Example + 9 (Q4): Multiple Aggregate without GROUP BY + key.Q4: SELECT SUM(ROUND(distance)) AS Total_Distance, + COUNT(stops) AS Count_of_Stops, + MIN(sourceairport) AS Min_of_Airport + FROM `travel-sample` + WHERE distance IS NOT MISSING;Results:[ + { + "Count_of_Stops": 24024, + "Min_of_Airport": "AAE", + "Total_Distance": 53538071 + } +]10. Expression in + Aggregate function

    +

    Aggregations with scalar expressions can be speeded up even if the index key does not + have the matching expression on the key. To use Aggregate Pushdown, use the + following syntax of the index and query + statement:CREATE INDEX idx_expr ON named_keyspace_ref (a,b,c); + +SELECT Aggregate_Function1(Expression(c)) +FROM named_keyspace_ref +WHERE a IS NOT MISSING +GROUP BY a,b;Example 10: + List the landmarks with the highest latitude.

    +

    Use the MAX() operator to find the highest landmark latitude in each + state, group the results by country and state, and + then sort in reverse order by the highest + latitudes.CREATE INDEX idx10 ON `travel-sample`(country, state, ABS(ROUND(geo.lat))) +WHERE type="landmark"; + +SELECT country, state, SUM(ABS(ROUND(geo.lat))) AS SumAbs_Latitude +FROM `travel-sample` +WHERE country IS NOT MISSING +AND type = "landmark" +GROUP BY country, state +ORDER BY SumAbs_Latitude DESC;The + Example 10 Explain Plan shows that Aggregates are executed by the indexer and is + detailed in the Aggregate Query Plan table:

    +

    { + "plan": { + "#operator": "Sequence", + "~children": [ + { + "#operator": "Sequence", + "~children": [ + { + "#operator": "IndexScan3", + "covers": [ + "cover ((`travel-sample`.`country`))", + "cover ((`travel-sample`.`state`))", + "cover (abs(round(((`travel-sample`.`geo`).`lat`))))", + "cover ((meta(`travel-sample`).`id`))", + "cover (sum(cover (abs(round(((`travel-sample`.`geo`).`lat`))))))" + ], + "filter_covers": { + "cover ((`travel-sample`.`type`))": "landmark" + }, + "index": "idx10", + "index_group_aggs": { + "aggregates": [ + { + "aggregate": "SUM", + "depends": [ + 2 + ], + "expr": "cover (abs(round(((`travel-sample`.`geo`).`lat`))))", + "id": 4, + "keypos": 2 + } + ], + "depends": [ + 0, + 1, + 2 + ], + "group": [ + { + "depends": [ + 0 + ], + "expr": "cover ((`travel-sample`.`country`))", + "id": 0, + "keypos": 0 + }, + { + "depends": [ + 1 + ], + "expr": "cover ((`travel-sample`.`state`))", + "id": 1, + "keypos": 1 + } +...Results:[ + { + "SumAbs_Latitude": 117513, + "country": "United Kingdom", + "state": null + }, + { + "SumAbs_Latitude": 68503, + "country": "United States", + "state": "California" + }, + { + "SumAbs_Latitude": 10333, + "country": "France", + "state": "Île-de-France" + }, +...11. SUM, COUNT, + MIN, MAX, or AVG Aggregate functions

    +

    Currently, the only aggregate functions that are supported are SUM(), COUNT(), MIN(), + MAX(), and AVG() with or without the DISTINCT modifier.

    +

    To use Aggregate Pushdown, use the below syntax of the index and query + statement:CREATE INDEX idx_expr ON named_keyspace_ref (a,b,c,d); + +SELECT Aggregate_Function(a), Aggregate_Function(b), + Aggregate_Function(c), Aggregate_Function(d) +FROM named_keyspace_ref +WHERE a IS NOT MISSING +GROUP BY a;Example + 11:CREATE INDEX idx11 ON `travel-sample`(ROUND(geo.lat), geo.alt, city, ROUND(geo.lon)) + WHERE type = "airport"; + +SELECT MIN(ROUND(geo.lat)) AS Min_Lat, + SUM(geo.alt) AS Sum_Alt, + COUNT(city) AS Count_City, + MAX(ROUND(geo.lon)) AS Max_Lon +FROM `travel-sample` +WHERE geo.lat IS NOT MISSING +AND type = "airport" +GROUP BY (ROUND(geo.lat)) +ORDER BY (ROUND(geo.lat)) DESC;Results:[ + { + "Count_City": 1, + "Max_Lon": 43, + "Min_Lat": 72, + "Sum_Alt": 149 + }, + { + "Count_City": 3, + "Max_Lon": -157, + "Min_Lat": 71, + "Sum_Alt": 120 + }, + { + "Count_City": 6, + "Max_Lon": -144, + "Min_Lat": 70, + "Sum_Alt": 292 + }, +...12. DISTINCT + aggregates

    +

    There are four cases when DISTINCT aggregates can use this feature:

      +
    1. If the DISTINCT aggregate is on the leading GROUP BY key(s).
      2. +
    2. If the DISTINCT aggregate is on the leading GROUP By key(s) + 1 (the + immediate next key).
      3. +
    3. If the DISTINCT aggregate is on a constant expression (GROUP BY can be on + any key).
      4. +
    4. If there is no GROUP BY and the DISTINCT aggregate is on the first key only + or in a constant expression.
      5. +
    To use Aggregate Pushdown, use one of the following syntaxes of the index and + query statements:

    +

    Case #1: If the DISTINCT aggregate is on the leading GROUP BY key(s).CREATE INDEX idx_expr ON named_keyspace_ref (a, b, c);

    + +
    Syntax A:
    +
    + SELECT SUM(DISTINCT a) +FROM named_keyspace_ref +WHERE a IS NOT MISSING +GROUP BY a; +
    +     +
    + +
    Syntax B:
    +
    + SELECT COUNT(DISTINCT a), SUM(DISTINCT b) +FROM named_keyspace_ref +WHERE a IS NOT MISSING +GROUP BY a, b; +
    +     +
    Example 12-1 (A): A DISTINCT aggregate on the leading GROUP BY + key(s).CREATE INDEX idx12_1 ON `travel-sample`(ROUND(geo.lat), ROUND(geo.lon), country) +WHERE type = "airport"; + +SELECT SUM(DISTINCT ROUND(geo.lat)) AS Sum_Lat +FROM `travel-sample` +WHERE geo.lat IS NOT MISSING +AND type = "airport" +GROUP BY ROUND(geo.lat);Results:[ + { + "Sum_Lat": 27 + }, + { + "Sum_Lat": 36 + }, + { + "Sum_Lat": 71 + }, +...Example + 12-1 (B): A DISTINCT aggregate on the leading GROUP BY + key(s).SELECT COUNT(DISTINCT ROUND(geo.lat)) AS Count_Lat, + SUM(DISTINCT ROUND(geo.lon)) AS Sum_Lon +FROM `travel-sample` +WHERE geo.lat IS NOT MISSING +AND type = "airport" +GROUP BY ROUND(geo.lat), ROUND(geo.lon);Results:[ + { + "Count_Lat": 1, + "Sum_Lon": -166 + }, + { + "Count_Lat": 1, + "Sum_Lon": -107 + }, + { + "Count_Lat": 1, + "Sum_Lon": -159 + }, +...Case + #2: If the DISTINCT aggregate is on the leading GROUP BY key(s) + 1 (the + next key)CREATE INDEX idx_expr ON named_keyspace_ref (a, b, c);
    + +
    Syntax A:
    +
    + SELECT SUM(DISTINCT b) +FROM named_keyspace_ref +WHERE a IS NOT MISSING +GROUP BY a; +
    +     +
    + +
    Syntax B:
    +
    + SELECT COUNT(DISTINCT c) +FROM named_keyspace_ref +WHERE a IS NOT MISSING +GROUP BY a, b; +
    +     +
    Example 12-2 (A): A DISTINCT aggregate on the leading GROUP BY key(s) + + 1 (the next + key).CREATE INDEX idx12_2 ON `travel-sample`(country, ROUND(geo.lat), ROUND(geo.lon)) +WHERE type = "airport"; + +SELECT COUNT(DISTINCT country) AS Count_Country, + SUM(DISTINCT ROUND(geo.lat)) AS Sum_Lat +FROM `travel-sample` +WHERE country IS NOT MISSING +AND type = "airport" +GROUP BY country;Results:[ + { + "Count_Country": 1, + "Sum_Lat": 483 + }, + { + "Count_Country": 1, + "Sum_Lat": 2290 + }, + { + "Count_Country": 1, + "Sum_Lat": 591 + } +]Example + 12-2 (B): A DISTINCT aggregate on the leading GROUP BY key(s) + 1 (the next + key)SELECT COUNT(DISTINCT country) AS Count_Country, + SUM(DISTINCT ROUND(geo.lat)) AS Sum_Lat, + COUNT(DISTINCT ROUND(geo.lon)) AS Count_Lon +FROM `travel-sample` +WHERE country IS NOT MISSING +AND type = "airport" +GROUP BY country, ROUND(geo.lat);Results:[ + { + "Count_Country": 1, + "Count_Lon": 16, + "Sum_Lat": 483 + }, + { + "Count_Country": 1, + "Count_Lon": 103, + "Sum_Lat": 2290 + }, + { + "Count_Country": 1, + "Count_Lon": 13, + "Sum_Lat": 591 + } +]Case + #3: If the DISTINCT aggregate is on a constant expression (GROUP BY can be + on any + key)CREATE INDEX idx_expr ON named_keyspace_ref (a, b, c); + +SELECT a, COUNT(DISTINCT 1) +FROM named_keyspace_ref +WHERE a IS NOT MISSING +GROUP BY b;The + results will be pre-aggregated if the GROUP BY key is + non-leading, as in this case and example.Example 12-3: A DISTINCT + aggregate on a constant expression (GROUP BY can be on any + key)CREATE INDEX idx12_3 ON `travel-sample`(country, geo.lat, geo.lon) +WHERE type = "airport"; + +SELECT MIN(country) AS Min_Country, + COUNT(DISTINCT 1) AS Constant_Value, + MIN(ROUND(geo.lon)) AS Min_Logitude +FROM `travel-sample` +WHERE country IS NOT MISSING +AND type = "airport" +GROUP BY geo.lat;Results:[ + { + "Constant_Value": 1, + "Min_Country": "United States", + "Min_Longitude": -75 + }, + { + "Constant_Value": 1, + "Min_Country": "United States", + "Min_Longitude": -169 + }, + { + "Constant_Value": 1, + "Min_Country": "United States", + "Min_Longitude": -165 + }, +...Case + #4: If the DISTINCT aggregate is on the first key only or in a constant + expression, and there is no GROUP BY + clauseCREATE INDEX idx_expr ON named_keyspace_ref (a, b, c); + +Q1: SELECT SUM(DISTINCT a) + FROM named_keyspace_ref; /* ok */ + +Q2: SELECT COUNT(DISTINCT 1) + FROM named_keyspace_ref; /* ok */ + +Q3: SELECT SUM(DISTINCT c) + FROM named_keyspace_ref; /* not ok */All + other cases of DISTINCT pushdown will return an error.

    +

    Example 12-4: A DISTINCT aggregate on the first key only or in a constant + expression, and there is no GROUP BY + clause.CREATE INDEX idx12_4 ON `travel-sample`(geo.alt, geo.lat, geo.lon) +WHERE type = "airport"; + +Q1: SELECT SUM(DISTINCT ROUND(geo.alt)) AS Sum_Alt + FROM `travel-sample` + WHERE geo.alt IS NOT MISSING + AND type = "airport";Results:[ + { + "Sum_Alt": 1463241 + } +]Another + query with index + idx12_4:Q2: SELECT COUNT(DISTINCT 1) AS Const_expr + FROM `travel-sample` + WHERE type = "airport";Results:[ + { + "Const_expr": 1 + } +]Another + query with index idx12_4 but will not pushdown the aggregate to the + indexer:Q3: SELECT SUM(DISTINCT ROUND(geo.lon)) AS Sum_Lon + FROM `travel-sample` + WHERE geo.alt IS NOT MISSING + AND type = "airport";Results:[ + { + "Sum_Lon": -11412 + } +]13. HAVING with an + aggregate function inside

    +

    To use Aggregate Pushdown when a HAVING clause has an aggregate function inside, use + the following syntax of index and query + statement:CREATE INDEX idx_expr ON named_keyspace_ref (k0, k1); + +SELECT k0, COUNT(k1) +FROM named_keyspace_ref +WHERE k0 IS NOT MISSING +GROUP BY k0 +HAVING Aggregate_Function(k1);Example + 13: HAVING with an aggregate function inside.

    +

    List the cities that have more than 180 + landmarks.CREATE INDEX idx13 ON `travel-sample` (city, name) +WHERE type = "landmark"; + +SELECT city AS City, COUNT(DISTINCT name) AS Landmark_Count +FROM `travel-sample` +WHERE city IS NOT MISSING +AND type = "landmark" +GROUP BY city +HAVING COUNT(DISTINCT name) > 180;Results:[ + { + "City": "London", + "Landmark_Count": 443 + }, + { + "City": "Los Angeles", + "Landmark_Count": 284 + }, + { + "City": "San Diego", + "Landmark_Count": 197 + }, + { + "City": "San Francisco", + "Landmark_Count": 797 + } +]14. LETTING with an + aggregate function inside

    +

    To use Aggregate Pushdown when a LETTING clause has an aggregate function inside, use + the following syntax of the index and query + statement.CREATE INDEX idx_expr ON named_keyspace_ref (k0, k1); + +SELECT k0, COUNT(k1) +FROM named_keyspace_ref +WHERE k0 IS NOT MISSING +GROUP BY k0 +LETTING var_expr = Aggregate_Function(k1) +HAVING var_expr;Example + 14: LETTING with an aggregate function inside.

    +

    List cities that have more than half of all + landmarks.CREATE INDEX idx14 ON `travel-sample` (city, name) +WHERE type = "landmark"; + +SELECT city AS City, COUNT(DISTINCT name) AS Landmark_Count +FROM `travel-sample` +WHERE city IS NOT MISSING +AND type = "landmark" +GROUP BY city +LETTING MinimumThingsToSee = COUNT(DISTINCT name) +HAVING MinimumThingsToSee > 180;Results:[ + { + "City": "London", + "Landmark_Count": 443 + }, + { + "City": "Los Angeles", + "Landmark_Count": 284 + }, + { + "City": "San Diego", + "Landmark_Count": 197 + }, + { + "City": "San Francisco", + "Landmark_Count": 797 + } +] +

    +
    +
    + Limitations +

    The following are currently not supported and not pushed to the indexer:

      +
    • HAVING or LETTING clauses, unless there is + an aggregate function inside.
      • +
    • ORDER BY clauses.
      • +
    • ARRAY_AGG() or any facility to add new Aggregate function, + such as Median.
      • +
    • LIMIT pushdown with GROUP BY on + non-leading keys.
      • +
    • OFFSET pushdown with GROUP BY on + non-leading keys.
      • +
    • A subquery in a GROUP BY or Aggregate pushdown.
      • +

    +

    Aggregate Comparison + + + + + + + + + Item + Aggregate on Non-Array Index Field + Aggregate on Array Index Field + DISTINCT Aggregate on Non-Array Index Field + DISTINCT Aggregate on Array Index Field + + + + + Supports MIN() and + MAX() + ✓ + ✓ + - + - + + + Supports SUM() and + COUNT() + - + ✓ + ✓ + ✓ + + + Supports AVG() + ✓ + ✓ + ✓ + ✓ + + + Supports ARRAY_AGG() + - + - + - + - + + + +

    +
    +
    Appx 1 - Query Plan Fields

    Consider the + example:EXPLAIN SELECT type, COUNT(type) +FROM `travel-sample` +WHERE type IS NOT MISSING +GROUP BY type;In + the query plan:

      +
    • plan.`~children`[0].covers shows that the index covers the + query.
      • +
    • plan.`~children`[0].index_group_aggs shows the aggregation + and groupings done by the indexer.
      • +
    • index_group_aggs object has details on the aggregate, index + key position, expression dependency, and group expressions handled by the + indexer. This object is present in the plan only when the indexer handles + the grouping and aggregation. + + + + + + + Item Name + Description + Explain Text in This Example + + + + + aggregates + Array of Aggregate objects, and each object + represents one aggregate function. The absence of this + item means there is no Aggregate function. + aggregates + + + ... aggregate + Aggregate operation. + COUNT + + + ... depends + List of index key positions the GROUP BY expression + depends on, starting with 0. + 0

      (because it's the 1st + item)

       +       + + ... expr + Group expression or an aggregate expression. + "cover + ((`travel-sample`.`type`))" + + + ... id + Unique ID given internally and will be used in + index_projection + 2 + + + ... keypos + Key Position to use the Index expr or the query + expr.
        +
      • A value > -1 means the group key exactly matches + the corresponding index keys, where 0 is the 1st + index key.
        • +
      • A value of -1 means the group key does not match + the index key and uses the query expression + instead.
        • +
      + 0

      (because it matches the 1st + index key)

       +       + + depends + List of index key positions the GROUP BY expression + depends on, starting with 0. + 0

      (because it's the 1st + item)

       +       + + group + Array of GROUP BY objects, and each object represents + one group key. The absence of this item means there is + no GROUP BY clause. + group + + + ... depends + Index key position of a single GROUP BY + expression + 0

      (because it's the 1st GROUP BY + key)

       +       + + ... expr + Single GROUP BY expression. + "cover + ((`travel-sample`.`type`))" + + + ... id + Unique ID given internally and will be used in + index_projection + 0 + + + ... keypos + Key Position to use the Index expr or the query + expr.
        +
      • A value > -1 means the group key exactly matches + the corresponding index keys, where 0 is the 1st + index key.
        • +
      • A value of -1 means the group key does not match + the index key and uses the query expression + instead.
        • +
      + 0

      (because it matches the 1st key + in the index expression)

       +       + + +
      { + "plan": { + "#operator": "Sequence", + "~children": [ + { + "#operator": "IndexScan3", + "covers": [ + "cover ((`travel-sample`.`name`))", + "cover ((meta(`travel-sample`).`id`))", + "cover (count(1))" + ], + "index": "idx_name", + "index_group_aggs": { + "aggregates": [ + { + "aggregate": "COUNT", + "expr": "1", + "id": 2, + "keypos": -1 + } + ], + "depends": [ + 0 + ], + "group": [ + { + "depends": [ + 0 + ], + "expr": "cover ((`travel-sample`.`name`))", + "id": 0, + "keypos": 0 + } + ] + }, + "index_id": "5dfe130db88b4ec", + "index_projection": { + "entry_keys": [ + 0, + 2 + ] + }, + "keyspace": "travel-sample", + "namespace": "default", + "spans": [ + { + "exact": true, + "range": [ + { + "inclusion": 1, + "low": "null" + } + ] + } + ], + "using": "gsi" + }, + { + "#operator": "Parallel", + "~child": { + "#operator": "Sequence", + "~children": [ + { + "#operator": "InitialProject", + "result_terms": [ + { + "expr": "cover ((`travel-sample`.`name`))" + }, + { + "expr": "cover (count(1))" + } + ] + }, + { + "#operator": "FinalProject" + } + ] + } + } + ] + }, + "text": "select name, count(1)\nfrom `travel-sample` use index (idx_name)\nwhere name is not missing\ngroup by name;" +}
      • +
    When the index_group_aggs section is present, it means + that the query is using Index Aggregations.

    GROUP BY Query Plan + + + + + + + Item Name + Description + EXPLAIN Text in Example #1 (GROUP BY) + + + + + aggregates + Array of Aggregate objects, and each object represents one + aggregate function. The absence of this item means there is no + Aggregate function. + aggregates + + + ... aggregate + Aggregate operation. + MAX + + + ... depends + List of index key positions the GROUP BY expression depends on, + starting with 0. + 2

    (because it's the 3rd item)

     +     + + ... expr + Group expression or an aggregate expression. + round(cover (((`travel-sample`. + `geo`).`lat`))) + + + ... id + Unique ID given internally and will be used in + index_projection + 4 + + + ... keypos + Key Position to use the Index expr or the query expr.
      +
    • A value > -1 means the group key exactly matches the + corresponding index keys, where 0 is the 1st index key.
      • +
    • A value of -1 means the group key does not match the index + key and uses the query expression instead.
      • +
    + -1

    (because the index has the field + geo.lat but the query adds the + ROUND() function to + geo.lat)

     +     + + depends + List of index key positions the GROUP BY expression depends on, + starting with 0. + 0, 1, 2 + + + group + Array of GROUP BY objects, and each object represents one group + key. The absence of this item means there is no GROUP BY + clause. + group + + + ... depends + Index key position of a single GROUP BY expression, starting with + 0. + 0

    (because it's the 1st GROUP BY + key)

     +     + + ... expr + Single GROUP BY expression. + `travel-sample`.`country` + + + ... id + Unique ID given internally and will be used in + index_projection. + 0 + + + ... keypos + Key Position to use the Index expr or the query expr.
      +
    • A value > -1 means the group key exactly matches the + corresponding index keys, where 0 is the 1st index key.
      • +
    • A value of -1 means the group key does not match the index + key and uses the query expression instead.
      • +
    + 0

    (because it matches the first key in the + index expression)

     +     + + +

    The Query Plan sections of an Aggregate pushdown are slightly different than + those used in a GROUP BY.

    Aggregate Query + Plan + + + + + + + Item Name + Description + EXPLAIN Text in Example #10 (Aggregate) + + + + + aggregates + Array of Aggregate objects, and each object represents one + aggregate function. The absence of this item means there is no + Aggregate function. + aggregates + + + ... aggregate + Aggregate operation. + SUM + + + ... depends + List of index key positions the GROUP BY expression depends + on, starting with 0. + 2

    (because it's the 3rd item)

     +     + + ... expr + Group expression or an aggregate expression. + "abs(round(cover + (((`travel-sample`.`geo`).`lat`))))" + + + ... id + Unique ID given internally and will be used in + index_projection + 4 + + + ... keypos + Key Position to use the Index expr or the query expr.
      +
    • A value > -1 means the group key exactly matches the + corresponding index keys, where 0 is the 1st index + key.
      • +
    • A value of -1 means the group key does not match the + index key and uses the query expression instead.
      • +
    + 2

    (because the query's 3rd key exactly + matches the index's 3rd key)

     +     + + depends + List of index key positions the GROUP BY expression depends + on, starting with 0. + 0, 1, 2 + + + group + Array of GROUP BY objects, and each object represents one + group key. The absence of this item means there is no GROUP BY + clause. + group + + + ... depends + Index key position of a single GROUP BY expression, starting + with 0. + 0

    (because it's the 1st GROUP BY + key)

     +     + + ... expr + Single GROUP BY expression. + "cover ((`travel-sample`.`state`))" + + + ... id + Unique ID given internally and will be used in + index_projection. + 0 + + + ... keypos + Key Position to use the Index expr or the query expr.
      +
    • A value > -1 means the group key exactly matches the + corresponding index keys, where 0 is the 1st index + key.
      • +
    • A value of -1 means the group key does not match the + index key and uses the query expression instead.
      • +
    + 0

    (because it matches the 1st key in the + index expression)

     +     + + ... depends + Index key position of a single GROUP BY expression, starting + with 0. + 0

    (because it's the 1st GROUP BY + key)

     +     + + ... expr + Single GROUP BY expression. + "cover ((`travel-sample`.`state`))" + + + ... id + Unique ID given internally and will be used in + index_projection. + 1 + + + ... keypos + Key Position to use the Index expr or the query expr.
      +
    • A value > -1 means the group key exactly matches the + corresponding index keys, where 0 is the 1st index + key.
      • +
    • A value of -1 means the group key does not match the + index key and uses the query expression instead.
      • +
    + 1

    (because it matches the 2nd key in the + index expression)

     +     + + +

    + +     diff --git a/content/n1ql/n1ql-language-reference/groupby.dita b/content/n1ql/n1ql-language-reference/groupby.dita index 42925d8d1a..6c5f067a97 100644 --- a/content/n1ql/n1ql-language-reference/groupby.dita +++ b/content/n1ql/n1ql-language-reference/groupby.dita @@ -1,61 +1,222 @@ - - + + GROUP BY clause - -

    group-by-clause:

    - GROUP BY expression [ , expression ]* [ letting-clause ] - [having-clause] ] | [letting-clause] -

    letting-clause:

    - LETTING alias = expression [(, alias = expression) ]* - -

    having-clause:

    - HAVING condition -

    GROUP BY allows you to organize a result set by one or more expressions. It can be used with - aggregate functions to group the result set. The optional LETTING and HAVING clauses follow - the GROUP BY clause.

    -

    Example

    - -SELECT relation, COUNT(*) AS count - FROM tutorial - GROUP BY relation - -

    Results

    - -{ - "results": [ - { - "count": 2, - "relation": "friend" - }, - { - "count": 1, - "relation": "coworker" - }, - { - "count": 1, - "relation": "parent" - }, - { - "count": 2, - "relation": "cousin" - } - ] -} - - -

    Example

    - SELECT title, type, COUNT(*) AS count FROM catalog GROUP BY type -

    LETTING Clause

    - -LETTING alias = expression [(, alias = expression) ]* - -

    Aliases in the LETTING clause create new names that can be referred to in the HAVING, SELECT, - and ORDER BY clauses.

    -

    HAVING Clause

    - HAVING condition -

    The HAVING clause optionally follows the GROUP BY clause. It filters a stream of variable - bindings, retaining those bindings for which the truth value of the HAVING clause is TRUE and - discarding the bindings for which this truth value is FALSE.

    -     -     + The GROUP BY clause arranges aggregate values into groups, based on one more + fields. + +

    (Introduced in Couchbase Server 4.0)

    +
    + Purpose +

    Use the GROUP BY clause to arrange aggregate values into groups of one or more fields. This + GROUP BY clause follows the WHERE clause and precedes + the optional LETTING, HAVING, and ORDER + BY clauses.

    +
    +
    + Syntax +

    +

    + GROUP BY expr [, expr2 ]* + [ LETTING alias = expr [, alias2 = expr2 ]* ] + [ HAVING cond ] +| +LETTING alias = expr [, alias2 = expr2 ]* +

    +
    +
    + Arguments +

    +

    + +
    expr
    +
    [At least one is required] String or expression representing the aggregate function or fields to group together.
    +     +
    +
    + +
    LETTING letting-clause
    +
    [Optional] Stores the result of a sub-expression in order to use it in subsequent + clauses.
    +
    +
    + +
    alias
    +
    String or expression representing the name of the clause to be referred + to.
    +     +
    +
    + +
    expr
    +
    String or expression representing the value of the LETTING + alias variable.
    +     +
    +
    +     +
    +
    + +
    HAVING having-clause
    +
    [Optional] To return items where aggregate + values meet the specified conditions.
    +
    +
    + +
    cond
    +
    String or expression representing the clause of aggregate values.
    +     +
    +
    +     +
    +

    +
    +
    + Limitations +

    GROUP BY works only on a group key or aggregate function.

    +
    +
    + Examples +

    Example 1: Group the unique landmarks by city and list the top 4 cities with the most + landmarks in descending order.

    + SELECT city City, COUNT(DISTINCT name) LandmarkCount +FROM `travel-sample` +WHERE type = "landmark" +GROUP BY city +ORDER BY LandmarkCount DESC +LIMIT 4; +

    Results:

    + [ + { + "City": "San Francisco", + "LandmarkCount": 797 + }, + { + "City": "London", + "LandmarkCount": 443 + }, + { + "City": "Los Angeles", + "LandmarkCount": 284 + }, + { + "City": "San Diego", + "LandmarkCount": 197 + } +] +

    Example 2: Use LETTING to find cities that have a minimum number of + things to see.

    + SELECT city City, COUNT(DISTINCT name) LandmarkCount +FROM `travel-sample` +WHERE type = "landmark" +GROUP BY city +LETTING MinimumThingsToSee = 400 +HAVING COUNT(DISTINCT name) > MinimumThingsToSee; +

    Results:[ + { + "City": "London", + "LandmarkCount": 443 + }, + { + "City": "San Francisco", + "LandmarkCount": 797 + } +]

    +

    Example 3: Use HAVING to specify cities that have more than 180 + landmarks.

    +

    + SELECT city City, COUNT(DISTINCT name) LandmarkCount +FROM `travel-sample` +WHERE type = "landmark" +GROUP BY city +HAVING COUNT(DISTINCT name) > 180; +

    +

    Results:

    + [ + { + "City": "London", + "LandmarkCount": 443 + }, + { + "City": "Los Angeles", + "LandmarkCount": 284 + }, + { + "City": "San Francisco", + "LandmarkCount": 797 + }, + { + "City": "San Diego", + "LandmarkCount": 197 + } +] + The above HAVING clause must use the aggregate function + COUNT instead of its alias LandmarkCount. +

    Example 4: Use HAVING to specify landmarks that begin with an "S" or + higher.

    +

    + SELECT city City, COUNT(DISTINCT name) LandmarkCount +FROM `travel-sample` +WHERE type = "landmark" +GROUP BY city +HAVING city > "S"; +

    +

    138 Results in 150ms:

    + [ + { + "City": "Santa Barbara", + "LandmarkCount": 53 + }, + { + "City": "San Francisco", + "LandmarkCount": 797 + }, + { + "City": "Stable Yd", + "LandmarkCount": 1 + }, + { + "City": "Wembley", + "LandmarkCount": 1 + }, +... +

    Example 4b: Using WHERE yields the same results as + HAVING, however, WHERE is faster.

    + SELECT city City, COUNT(DISTINCT name) LandmarkCount +FROM `travel-sample` +WHERE type = "landmark" +AND city > "S" +GROUP BY city +

    138 Results in + 94ms:[ + { + "City": "San Luis Obispo", + "LandmarkCount": 1 + }, + { + "City": "Twentynine Palms", + "LandmarkCount": 1 + }, + { + "City": "Westlake Village", + "LandmarkCount": 1 + }, + { + "City": "Surrey", + "LandmarkCount": 1 + }, +...

    + The WHERE clause is faster because WHERE gets + processed before any GROUP BY and doesn't have access to aggregated + values. HAVING gets processed after + GROUP BY and is used to constrain the resultset to only those with + aggregated values. +
    + + diff --git a/content/n1ql/n1ql-language-reference/hints.dita b/content/n1ql/n1ql-language-reference/hints.dita index 88be8893e6..60c5f67a64 100644 --- a/content/n1ql/n1ql-language-reference/hints.dita +++ b/content/n1ql/n1ql-language-reference/hints.dita @@ -1,30 +1,85 @@ - - + + USE INDEX clause - The USE INDEX clause supplies index hints that specify an index to use as part of the - query execution. The query engine attempts to use the specified index if the index is - applicable for the query. - -

    use-index-clause:

    - USE INDEX ( index-ref [ , index-ref ]* ) -

    index-ref:

    - index-name [ index-using ] -

    index-name:

    - identifier -

    index-using:

    - USING ( VIEW | GSI ) - -

    Here's an example that shows how to use index hints. The example creates a GSI index called - beer_abv, and then selects the name and - abv fields, specifying that the query engine use the - beer_abv index.

    - CREATE INDEX beer_abv ON `beer-sample`(abv) USING GSI; + The USE INDEX clause specifies the index to use in the query + execution. + +

    (Introduced in Couchbase Server 3.0)

    +
    + Purpose +

    Use the USE INDEX clause to specify which index to use as part of + the query execution. The query engine attempts to use the specified index if the + index is applicable for the query.

    +
    +
    + Prerequisites +

    For you to select data from a document or keyspace, you must have the + query_select privilege on the document or keyspace. For + more details about user roles, see .

    +
    +
    + Syntax +

    USE INDEX ( index-ref [ , index-ref2 ]* ) [ USING ( VIEW | GSI ) ]

    +
    +
    + Arguments +

    +

    + +
    index-ref
    +
    [Required] String or expression representing the index or indexes to be + used for the query.
    +     +
    +
    + +
    USING
    +
    [Optional; default is USING GSI] Specifies which index + form to use.
    +
    +
    + +
    VIEW
    +
    The legacy index form, which lives on the data node.
    +     +
    +
    + +
    GSI
    +
    The newer and faster Global Secondary Index form, which + lives on an index node and can possibly be separate from a + data node.
    +     +
    +
    +     +
    +

    +
    +
    + Examples +

    Example 1a: Use an existing index with GSI in a query.

    +

    Create an index of airlines and destination airports, and then use it in a query for + flights originating in San Francisco.

    + CREATE INDEX idx_destinations +ON `travel-sample` (airlineid, airline, destinationairport) +WHERE type="route"; -SELECT name, abv FROM `beer-sample` USE INDEX (beer_abv USING GSI) WHERE abv > 10; -

    The following example shows how to perform a similar query with a view index:

    - CREATE INDEX beer_abv ON `beer-sample`(abv) USING VIEW; +SELECT airlineid, airline, sourceairport, destinationairport +FROM `travel-sample` USE INDEX (idx_destinations USING GSI) +WHERE sourceairport = "SFO"; +

    Example 1b: Use an existing index with VIEW in a query.

    +

    The usage of VIEW is identical as that of GSI.

    + CREATE INDEX idx_destinations +ON `travel-sample` (airlineid, airline, destinationairport) +WHERE type="route"; -SELECT name, abv FROM `beer-sample` USE INDEX (beer_abv USING VIEW) WHERE abv > 10; - - +SELECT airlineid, airline, sourceairport, destinationairport +FROM `travel-sample` USE INDEX (idx_destinations USING VIEW) +WHERE sourceairport = "SFO"; +
    + +     diff --git a/content/n1ql/n1ql-language-reference/images/CURL_Add_button.png b/content/n1ql/n1ql-language-reference/images/CURL_Add_button.png new file mode 100644 index 0000000000..3b574aa6bf Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/CURL_Add_button.png differ diff --git a/content/n1ql/n1ql-language-reference/images/CURL_Remove_button.png b/content/n1ql/n1ql-language-reference/images/CURL_Remove_button.png new file mode 100644 index 0000000000..106e8d6051 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/CURL_Remove_button.png differ diff --git a/content/n1ql/n1ql-language-reference/images/CURL_Whitelist.png b/content/n1ql/n1ql-language-reference/images/CURL_Whitelist.png new file mode 100644 index 0000000000..9308a1bc02 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/CURL_Whitelist.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_AnsiJoin-Ex4-BeerVisual1.png b/content/n1ql/n1ql-language-reference/images/FROM_AnsiJoin-Ex4-BeerVisual1.png new file mode 100755 index 0000000000..282f3b6f4b Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_AnsiJoin-Ex4-BeerVisual1.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_AnsiJoin-Ex4-BeerVisual2.png b/content/n1ql/n1ql-language-reference/images/FROM_AnsiJoin-Ex4-BeerVisual2.png new file mode 100755 index 0000000000..93f277f6be Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_AnsiJoin-Ex4-BeerVisual2.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_ansi-join_RR-clause_5.5.png b/content/n1ql/n1ql-language-reference/images/FROM_ansi-join_RR-clause_5.5.png new file mode 100755 index 0000000000..1b0a43d67b Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_ansi-join_RR-clause_5.5.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_ansi-nest_RR-clause_5.5.png b/content/n1ql/n1ql-language-reference/images/FROM_ansi-nest_RR-clause_5.5.png new file mode 100755 index 0000000000..d445507eff Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_ansi-nest_RR-clause_5.5.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_from-clause.png b/content/n1ql/n1ql-language-reference/images/FROM_from-clause.png new file mode 100755 index 0000000000..3283e47c13 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_from-clause.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_from-clause_4.0.png b/content/n1ql/n1ql-language-reference/images/FROM_from-clause_4.0.png new file mode 100755 index 0000000000..cf63e862a6 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_from-clause_4.0.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_from-clause_5.5.png b/content/n1ql/n1ql-language-reference/images/FROM_from-clause_5.5.png new file mode 100755 index 0000000000..99e7fd2057 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_from-clause_5.5.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_from-clause_5.5_RR.png b/content/n1ql/n1ql-language-reference/images/FROM_from-clause_5.5_RR.png new file mode 100755 index 0000000000..994707f234 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_from-clause_5.5_RR.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_index-join-clause.png b/content/n1ql/n1ql-language-reference/images/FROM_index-join-clause.png new file mode 100755 index 0000000000..0039a021ca Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_index-join-clause.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_index-nest_5.1_rr.png b/content/n1ql/n1ql-language-reference/images/FROM_index-nest_5.1_rr.png new file mode 100755 index 0000000000..331c825f6b Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_index-nest_5.1_rr.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_lookup-join-clause.png b/content/n1ql/n1ql-language-reference/images/FROM_lookup-join-clause.png new file mode 100755 index 0000000000..dd0ffe223d Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_lookup-join-clause.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_lookup-nest_4.0_RR.png b/content/n1ql/n1ql-language-reference/images/FROM_lookup-nest_4.0_RR.png new file mode 100755 index 0000000000..7760436ad4 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_lookup-nest_4.0_RR.png differ diff --git a/content/n1ql/n1ql-language-reference/images/FROM_unnest-clause_4.0_RR.png b/content/n1ql/n1ql-language-reference/images/FROM_unnest-clause_4.0_RR.png new file mode 100755 index 0000000000..97be4fe941 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/FROM_unnest-clause_4.0_RR.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_55execution.png b/content/n1ql/n1ql-language-reference/images/GBAP_55execution.png new file mode 100644 index 0000000000..e62d9d6a8c Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_55execution.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_BottomUp_GB_non-leading.png b/content/n1ql/n1ql-language-reference/images/GBAP_BottomUp_GB_non-leading.png new file mode 100644 index 0000000000..eb57bb0af5 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_BottomUp_GB_non-leading.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_Ex0_QP_after55.png b/content/n1ql/n1ql-language-reference/images/GBAP_Ex0_QP_after55.png new file mode 100644 index 0000000000..e3b5b7d831 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_Ex0_QP_after55.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_Ex0_QP_before55.png b/content/n1ql/n1ql-language-reference/images/GBAP_Ex0_QP_before55.png new file mode 100644 index 0000000000..84983abeea Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_Ex0_QP_before55.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_Ex10_VP.png b/content/n1ql/n1ql-language-reference/images/GBAP_Ex10_VP.png new file mode 100644 index 0000000000..83c6dc883d Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_Ex10_VP.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_Ex2A_EP.png b/content/n1ql/n1ql-language-reference/images/GBAP_Ex2A_EP.png new file mode 100644 index 0000000000..808a6f0ce6 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_Ex2A_EP.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_Ex4A_VP.png b/content/n1ql/n1ql-language-reference/images/GBAP_Ex4A_VP.png new file mode 100644 index 0000000000..5c2f41240e Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_Ex4A_VP.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_ExB_Plan.png b/content/n1ql/n1ql-language-reference/images/GBAP_ExB_Plan.png new file mode 100644 index 0000000000..879d214ae6 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_ExB_Plan.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_ExC_Plan.png b/content/n1ql/n1ql-language-reference/images/GBAP_ExC_Plan.png new file mode 100644 index 0000000000..65ab3d39f4 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_ExC_Plan.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_ExD2_Plan.png b/content/n1ql/n1ql-language-reference/images/GBAP_ExD2_Plan.png new file mode 100644 index 0000000000..2857ee0810 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_ExD2_Plan.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_ExD_Plan.png b/content/n1ql/n1ql-language-reference/images/GBAP_ExD_Plan.png new file mode 100644 index 0000000000..87f0cd200c Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_ExD_Plan.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_ExE_Plan.png b/content/n1ql/n1ql-language-reference/images/GBAP_ExE_Plan.png new file mode 100644 index 0000000000..c5f2587919 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_ExE_Plan.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_LeadingKey_OldMethod.png b/content/n1ql/n1ql-language-reference/images/GBAP_LeadingKey_OldMethod.png new file mode 100644 index 0000000000..8a8a6d65aa Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_LeadingKey_OldMethod.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_PosEx_QP.png b/content/n1ql/n1ql-language-reference/images/GBAP_PosEx_QP.png new file mode 100644 index 0000000000..560b8355ac Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_PosEx_QP.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_Unnest_All_ArrayKey.png b/content/n1ql/n1ql-language-reference/images/GBAP_Unnest_All_ArrayKey.png new file mode 100644 index 0000000000..2022454754 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_Unnest_All_ArrayKey.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_leadingkey_arraykey.png b/content/n1ql/n1ql-language-reference/images/GBAP_leadingkey_arraykey.png new file mode 100644 index 0000000000..e37a4a9e4f Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_leadingkey_arraykey.png differ diff --git a/content/n1ql/n1ql-language-reference/images/GBAP_pre55execution.png b/content/n1ql/n1ql-language-reference/images/GBAP_pre55execution.png new file mode 100644 index 0000000000..c3164667a2 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/GBAP_pre55execution.png differ diff --git a/content/n1ql/n1ql-language-reference/images/Union_Intersect_Except.png b/content/n1ql/n1ql-language-reference/images/Union_Intersect_Except.png new file mode 100644 index 0000000000..1363e56273 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/Union_Intersect_Except.png differ diff --git a/content/n1ql/n1ql-language-reference/images/Union_Intersect_Except_diagram+desc.png b/content/n1ql/n1ql-language-reference/images/Union_Intersect_Except_diagram+desc.png new file mode 100644 index 0000000000..1cae6af675 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/Union_Intersect_Except_diagram+desc.png differ diff --git a/content/n1ql/n1ql-language-reference/images/alter-index-clause.png b/content/n1ql/n1ql-language-reference/images/alter-index-clause.png new file mode 100644 index 0000000000..ca2855d15b Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/alter-index-clause.png differ diff --git a/content/n1ql/n1ql-language-reference/images/alter-index_action-clause.png b/content/n1ql/n1ql-language-reference/images/alter-index_action-clause.png new file mode 100644 index 0000000000..3058964c57 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/alter-index_action-clause.png differ diff --git a/content/n1ql/n1ql-language-reference/images/alter-index_node-clause.png b/content/n1ql/n1ql-language-reference/images/alter-index_node-clause.png new file mode 100644 index 0000000000..eb84628878 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/alter-index_node-clause.png differ diff --git a/content/n1ql/n1ql-language-reference/images/alter-index_servers_step1.png b/content/n1ql/n1ql-language-reference/images/alter-index_servers_step1.png new file mode 100644 index 0000000000..bc95477eb5 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/alter-index_servers_step1.png differ diff --git a/content/n1ql/n1ql-language-reference/images/alter-index_servers_step2.png b/content/n1ql/n1ql-language-reference/images/alter-index_servers_step2.png new file mode 100644 index 0000000000..10474d311c Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/alter-index_servers_step2.png differ diff --git a/content/n1ql/n1ql-language-reference/images/alterindex_CONFLICTED.xml b/content/n1ql/n1ql-language-reference/images/alterindex_CONFLICTED.xml new file mode 100644 index 0000000000..ddbc293d1c --- /dev/null +++ b/content/n1ql/n1ql-language-reference/images/alterindex_CONFLICTED.xml @@ -0,0 +1,307 @@ + + + + ALTER INDEX + The ALTER INDEX statement moves the placement of an existing index or replica among + different GSI nodes. + +

    (Introduced in Couchbase Server 5.5 Enterprise Edition)

    +
    + Purpose +

    Use the ALTER INDEX statement to change the placement of an existing + index or replica among different GSI nodes when you encounter any of the following + situations:

      +
    • An imbalance occurs due to a particular index growing faster than expected + and is needed on a different node.
      • +
    • An imbalance occurs due to a cluster of indexes being dropped on a single + node.
      • +
    • A machine is scheduled for removal, so its indexes need to move off its + current node.
      • +
    • The automated process of rebalancing does not give the expected + results.
      • +
    • Other types of scaling up or scaling down are needed.
      • +
    For example, if a node fails and you need to move it from node 172.23.130.24 + to node 172.23.130.25, then you can simply execute the following code + with your own index names and + addresses:CREATE INDEX idx1 ON `travel-sample`(id, airline) WHERE type="route"; + +ALTER INDEX `travel-sample`.idx1 +WITH {"action":"move","nodes": ["172.23.120.25:8091"]}The + ALTER INDEX operation cannot start if a node is undergoing + a rebalance.The ALTER INDEX move operation is asynchronous. As soon + as the move alter index command is executed, the command returns. If there is no + error in the input, the move operation can be tracked through the console UI and + any error can be found in the Console logs and Indexer logs.If a node + goes down while an ALTER INDEX operation is in progress, then the index would + rollback to its original node (not affecting queries) and a notification would + appear.

    +
    +
    + Prerequisites +

    Only users with the suitable RBAC role of Administrator are allowed + to run the ALTER INDEX directive.

    +

    This directive is applicable only for Standard GSI (Plasma) and MOI Indexes; and + hence supported only for Enterprise Edition nodes. (Not Applicable for Forest + DB).

    +

    This statement is supported only from 5.5 onwards; if the cluster is in mix-mode (a + mix of 5.5 and previous versions), then this directive will not work.

    +

    ALTER INDEX will not work while the cluster is undergoing a + Rebalance.

    +
    +
    + Syntax +

    +

    +

    ALTER INDEX key_expr +[ USING GSI ] +WITH {"action":"action_name", "nodes": [ "node_expr" [, "node_expr2"]* }

    +
    +
    + Arguments +

    +

    + +
    key_expr
    +
    [Required] String representing the corresponding named keyspace + reference, with an optional index name for example + named_keyspace_ref.index_name.
    +     +
    +

    +
    + +
    USING GSI
    +
    [Optional. Default is "USING GSI"]
    +
    Uses the Global Secondary Index.
    +     +
    +
    + +
    WITH
    +
    [Required] Specifies the alter operation that needs to be performed.
    +
    +
    + +
    action
    +
    [Required] Reserved word for denoting the single + action_name operation to be + performed.
    +     +
    +
    + +
    action_name
    +
    +
    + +
    move
    +
    [Required] Moves only 1 index (or its replica) at a + time to a different node.The length of the + nodes array must be equal to the number of index + replicas.
    +     +
    +
    +     +
    +
    + +
    nodes
    +
    [Required] Reserved word for denoting the node list that + specifies the new destination nodes for the index and its + replicas.The full node list needs to be specified even + if only 1 replica needs to be moved.
    +     +
    +
    + +
    node_expr
    +
    [Required] String of the destination node address or + addresses.
    +     +
    +
    +     +
    +
    +
    + Return Value +

    If the ALTER INDEX succeeds, then:

      +
    • The Query Workbench will show { Results: [] }
      • +
    • The index progress will be visible on the UI.
      • +
    • After the movement is complete, the new indexes will begin to service query + scans.
      • +
    • The command line will display the new index nodes.
      • +
    If the ALTER INDEX fails, then:
      +
    • The original indexes will continue to service query scans.
      • +
    • The UI Log and Query Workbench will have the appropriate error message.
      • +
    • Some common errors include: + + + + + + Error Message + Possible Cause + + + + + GSI index xxxxxxxx not found + +
        +
      • Mistyped an index name
        • +
      +       +       + + Missing Node Information For Move + Index + +
        +
      • Mistyped "node" instead of + "nodes"
        • +
      • Mistyped punctuation or other item
        • +
      +       +       + + No Index Movement Required for Specified + Destination List + +
        +
      • Entered the current node instead of the target + node
        • +
      +       +       + + syntax error - at \",\" + +
        +
      • Missed a double-quote mark + (")
        • +
      +       +       + + Unable to find Index service for destination + xxx.xxx.xxx.xxx:8091 or destination is not part of + the cluster + +
        +
      • Address doesn't exist or was mistyped
        • +
      • Node isn't running
        • +
      • Node not properly added to the cluster
        • +
      +       +       + + Unsupported action value + +
        +
      • Mistyped the "action"
        • +
      +       +       + + +
      • +

    +
    +
    Examples

    When using the below examples, + make sure Couchbase Server 5.5 Enterprise Edition is already running on the named + nodes.

    Example 1: Move the def_faa index from one node to + another.

    Create a cluster of 3 nodes and then go to Settings > + Sample buckets to install the travel-sample bucket. The + indexes will then be installed in a round-robin fashion and distributed over the 3 + nodes. Then move the def_faa index from the first node + (192.168.10.10 in the screenshot) to the second node + (192.168.10.11 in the screenshot).

    ALTER INDEX `travel-sample`.def_faa +WITH {"action":"move","nodes": ["192.168.10.11:8091"]}You + should see:{ + "results": [] +}

    Example 2: Create and move an index replica from one node to + another.

    Create an index on node 192.168.10.10 with a replica on node + 192.168.10.11, then move its replica from node 192.168.10.11 to + 192.168.10.12.CREATE INDEX country_idx ON `travel-sample`(country, city) + WHERE type="route" USING GSI + WITH {"nodes":["192.168.10.10:8091", "192.168.10.11:8091"]}; + + +ALTER INDEX `travel-sample`.country_idx +WITH {"action":"move","nodes": ["192.168.10.10:8091", "172.23.120.12:8091"]}

    Example + 3: Moving multiple replicas.

    Create an index on node 192.168.10.10 with + replicas on nodes 192.168.10.11 and 192.168.10.12, then move the + replicas to nodes 192.168.10.13 and + 192.168.10.14.CREATE INDEX country_idx ON `travel-sample`(country, city) +WITH {"nodes": ["192.168.10.10:8091", "192.168.10.11:8091", "192.168.10.12:8091"]} + +ALTER INDEX `travel-sample`.country_idx +WITH {"action":"move","nodes": + ["192.168.10.10:8091", "192.168.10.13:8091", "192.168.10.14:8091"]}

    +
    Example 4: Removing an extra replica.

    If you + created an index on node 192.168.10.10 with replicas on nodes 192.168.10.11 and + 192.168.10.12 and later decided you didn't want the 2nd replica, then you'll need to + remove the index (which removes all replicas) and then re-create the index with only + one + replica.CREATE INDEX country_idx ON `travel-sample`(country, city) +WHERE type="route" USING GSI +WITH {"nodes":["192.168.10.10:8091", "192.168.10.11:8091", "192.168.10.12:8091"]}; + + +DROP INDEX `travel-sample`.country_idx; + +CREATE INDEX country_idx ON `travel-sample`(country, city) +WHERE type="route" USING GSI +WITH {"nodes":["192.168.10.10:8091", "192.168.10.11:8091"]}; +

    +

    +

    +

    +

    +
    + + + +     diff --git a/content/n1ql/n1ql-language-reference/images/create-partitioned-index-syntax.png b/content/n1ql/n1ql-language-reference/images/create-partitioned-index-syntax.png new file mode 100644 index 0000000000..3444911724 Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/create-partitioned-index-syntax.png differ diff --git a/content/n1ql/n1ql-language-reference/images/lookup-join-clause.png b/content/n1ql/n1ql-language-reference/images/lookup-join-clause.png new file mode 100644 index 0000000000..dd0ffe223d Binary files /dev/null and b/content/n1ql/n1ql-language-reference/images/lookup-join-clause.png differ diff --git a/content/n1ql/n1ql-language-reference/images/order-by-clause.png b/content/n1ql/n1ql-language-reference/images/order-by-clause.png old mode 100755 new mode 100644 index 3613ce897f..451fc25258 Binary files a/content/n1ql/n1ql-language-reference/images/order-by-clause.png and b/content/n1ql/n1ql-language-reference/images/order-by-clause.png differ diff --git a/content/n1ql/n1ql-language-reference/index-partitioning.dita b/content/n1ql/n1ql-language-reference/index-partitioning.dita new file mode 100644 index 0000000000..9ccbfd174a --- /dev/null +++ b/content/n1ql/n1ql-language-reference/index-partitioning.dita @@ -0,0 +1,571 @@ + + + + Index Partitioning + Index Partitioning enables you to increase aggregate query performance by dividing and spreading a large index of documents across multiple nodes, horizontally scaling out an index as needed. The system partitions the index across a number of index nodes using a hash partitioning strategy in a way that is transparent to queries. + +

    (Introduced in Couchbase Server 5.5 Enterprise Edition)

    +
    +

    Benefits of a partitioned index include:

      +
    • The ability to scale out horizontally as the index size increases.
      • +
    • Transparency to queries, requiring no change to existing queries.
      • +
    • Reduction of query latency for large aggregated queries since each partition can be scanned in parallel.
      • +
    • Provides a low-latency range query while allowing indexes to be scaled out as needed.
      • +

    +
    +
    + Create Partitioned Index +

    SyntaxCREATE INDEX idx_name ON bucket_name ( field_name [ , field_name2 ] * ) + PARTITION BY HASH partition_key_expr + [ WITH { with_expr [ , with_expr2 ] * } ];

    +
    +
    + Arguments +

    +

    + +
    idx_name
    +
    Identifier representing the + name of your index. The partitioned index name must unique within a + bucket.
    +     +
    +
    + +
    bucket_name
    +
    String representing the bucket to be indexed.
    +
    + If there is a hyphen (-) inside the bucket name, backticks (` `) + are needed around the bucket name. +
    +     +
    +
    + +
    ( field_name )
    +
    String of one or more field names (comma-separated) to be included in + the index.
    +
    + The field name(s) must be enclosed inside parentheses. +
    +     +
    +
    + +
    ( partition_key_expr )
    +
    One or more fields or an expression of one or more fields representing + the partition keys.This expression must be enclosed inside + parentheses.
    +     +
    +
    + +
    WITH { }
    +
    [Optional] The reserved word to be used with one or more of the + following arguments.
    +     +
    +
    + +
    with_expr
    +
    Required if WITH {} is present. The with_expr must contain at least one of the following arguments.
    +
    +
    + +
    "num_parition": partition_int
    +
    The integer partition_int defines the number of + partitions to divide into. The default value is 8. For + details and examples, see Number of Index Partitions.
    +     +
    +
    + +
    "nodes": ["###.###.###.###:####", ...]
    +
    The node list to restrict the set of nodes available for + placement, separated by commas. For details and examples, + see Node Placement.
    +     + +
    "defer_build": true | false
    +
    When defer_build is set to true, the index + creation operation queues the task for building the index + but immediately pauses the building of the index of type + GSI. Index building requires an expensive scan operation and + deferring building of an index with multiple indexes can + optimize the expensive scan operation. Administrators can + defer building multiple indexes and use the BUILD INDEX + statement to build multiple indexes efficiently with one + efficient scan of the bucket data. When + defer_build is set to false, the CREATE + INDEX operation queues the task for building the index and + immediately kicks off the building of the index of type + GSI.
    +     + +
    "num_replica": num_replica_num
    +
    The number of replicas of the partitioned index to create. + The index service will automatically distribute these + indexes amongst the index nodes in the cluster for load + balancing and high availability purposes. When creating an + index with replicas in this manner, the index service + attempts to distribute the replicas based on the server + groups in use in the cluster where possible. If + num_replica_num is not less than the + number of index nodes in the cluster, then the index + creation will fail.
    +     + +
    secKeySize: sec_int
    +
    The average length of the combined index keys. For details + and examples, see Sizing Hint.
    +     + +
    docKeySize: doc_int
    +
    The average length of the document key. For details and + examples, see Sizing Hint.
    +     + +
    arrSize: arr_int
    +
    The average length of the aray fields. For details and + examples, see Sizing Hint.
    +     +
    +
    +     +
    +

    +
    + +
    + Partition Keys +

    Partition keys are made up of one or more terms, with each term being the document + key, a document field, or an expression of document key or field. The partition keys + are hashed to generate a partition ID for each document. The partition ID is then + used to identify the partition in which the document's index keys would reside.

    +

    The partition keys should be immutable, that is, its values shouldn't change once the + document is created. For example, the 'travel-sample' field type almost never + changes and is therefore a good candidate for partition key. If the partition keys + have changed, then the corresponding document should be deleted and recreated with + the new partition keys.

    +

    Each term in the partition keys can be any JSON data type: number, string, boolean, + array, object, or NULL. If a term in the partition keys is missing in the document, + the term will have a N1QL MISSING value. Partition keys do not support N1QL array + expressions (for e.g.,ARRAY ... FOR ... + IN).

    +

    The following table lists some examples of partition keys.

    +

    + + + + + + + Partition Type + Example + + + + + The document key. + + CREATE INDEX idx ON `travel-sample`(country, airline, id) + PARTITION BY HASH(META().id); + + + + Any single or multiple immutable field in the defined + index. + + CREATE INDEX idx ON `travel-sample`(sourceairport,destinationairport, stops, airline, id) + PARTITION BY HASH(sourceairport,destinationairport); + + + + Any single or multiple immutable non-leading field in the + defined index. + + CREATE INDEX idx ON `travel-sample`(airline, sourceairport, destinationairport, stops, id) + PARTITION BY HASH(sourceairport, destinationairport); + + + + Any single or multiple immutable document field not defined + in the index. + + CREATE INDEX idx ON `travel-sample` (sourceairport, stops, airline, id) + PARTITION BY HASH (sourceairport, destinationairport) + + + + A function on the index fields, such as LOWER(), + LEAST(), GREATEST(), SUBSTR(), etc. + + CREATE INDEX idx ON `travel-sample`(LOWER(sourceairport), LOWER(destinationairport), stops, airline, id) + PARTITION BY HASH(LOWER(sourceairport), LOWER(destinationairport)); + + + + A complex expression on the index fields combining functions + and operators. + + CREATE INDEX idx ON `travel-sample`(POSITION(meta().id,'__')+2, destinationairport, sourceairport, stops, airline, id) + PARTITION BY HASH(POSITION(meta().id,'__')+2)); + + + + +
    +

    + +
    +
    Using Document Keys as Partition Key +

    The simplest way to create a partitioned index is to use the document key as the + partition key.

    +

    Example 1: Create a partitioned index with partition key being the document key

    + CREATE INDEX idx_pe1 ON `travel-sample`(country, airline, id) + PARTITION BY HASH(META().id); + +SELECT airline, id +FROM `travel-sample` +WHERE country="United States" +ORDER BY airline; +

    With meta().id as the partition key, the index keys are evenly + distributed among all the partitions. Every query will gather the qualifying index + keys from all the partitions.

    +
    +
    + Choosing Partition Keys for Range Query +

    An application has the option to choose the partition key that can minimize latency + on a range query for a partitioned index. For example, let's say a query has an + equality predicate based on the field sourceairport and + destinationairport. If the index is also partitioned by the + index keys on sourceairport and + destinationairport, then the query will only need to read a single + partition for the given pair of sourceairport and + destinationairport. In this case, the application can maintain + a low query latency while allowing the partitioned index to scale out as needed.

    +

    Example 2: Create a partitioned index with partition keys matching query equality + predicate

    + # Lookup all airlines with non-stop flights from SFO to JFK +CREATE INDEX idx_pe2 ON `travel-sample` (sourceairport, destinationairport, stops, airline, id) + PARTITION BY HASH (sourceairport, destinationairport); + +SELECT airline, id +FROM `travel-sample` +WHERE sourceairport=”SFO” AND +destinationairport=”JFK” AND +stops == 0 +ORDER BY airline; +

    The partition keys do not have to be the leading index keys in order to select + qualifying partitions As long as the leading index keys are provided along with the + partition keys in the predicate, the query engine can still select the qualifying + partitions for index scan. The following example scans a single partition with a + given pair of sourceairport and + destinationairport.

    +

    Example 3: Create a partitioned index with partition keys being non-leading index + keys

    + # Lookup all non-stop flights from SFO to JFK for the given airlines +CREATE INDEX idx_pe3 ON `travel-sample` (airline, sourceairport, destinationairport, stops, id) + PARTITION BY HASH (sourceairport, destinationairport); + +SELECT airline, id +FROM `travel-sample` +WHERE airline in [“UA”, “AA”] AND +sourceairport=”SFO” AND +destinationairport=”JFK” AND +stops == 0 +ORDER BY airline; +

    If the partition keys are basedon a N1QL expression, then the query predicate should + use the same expression for selecting qualifying partitions.

    +

    Example 4: Create a partitioned index with partition keys as expressions

    + # Case-insensitive lookup for all airlines with non-stop flights from SFO to JFK +CREATE INDEX idx_pe4 ON `travel-sample` (LOWER(sourceairport), LOWER(destinationairport), stops, airline, id) + PARTITION BY HASH (LOWER(sourceairport), LOWER(destinationairport)) + +SELECT airline, id +FROM `travel-sample` +WHERE LOWER(sourceairport)=”sfo” AND +LOWER(destinationairport)=”jfk” AND +stops == 0 +ORDER BY airline +

    As with equality predicate in the previous examples, the query engine can select + qualifying partitions using an IN clause with matching partitioned keys. The + following example scans at most three partitions with sourceairport + "SFO", "SJC", or "OAK".

    +

    Example 5: Create a partitioned index with partition keys matching query IN + clause

    + # Lookup for all airlines with non-stop flights from SFO, SJC, or OAK to JFK +CREATE INDEX idx_pe5 ON `travel-sample` (sourceairport, destinationairport, stops, airline, id) + PARTITION BY HASH (sourceairport, destinationairport); + +SELECT airline, id +FROM `travel-sample` +WHERE sourceairport in [”SFO”, “SJC”, “OAK”] AND +destinationairport=”JFK” AND +stops == 0 +ORDER BY airline; +

    As shown in the previous examples, in order to allow the query engine to select + qualifying partitions, the partition keys must be present as an equality predicate + in the query. The following query only has an equality predicate on + sourceairport and hence will not be able to select the + qualifying partitions without destinationairport. Consequently, + this query will gather qualifying index keys from all partitions.

    +

    Example 6: Create a partitioned index with non-matching query equality + predicate

    + # Lookup all airlines with non-stop flights from SFO +CREATE INDEX idx_pe6 ON `travel-sample` (sourceairport, destinationairport, stops, airline, id) + PARTITION BY HASH (sourceairport, destinationairport); + +SELECT airline, id +FROM `travel-sample` +WHERE sourceairport=”SFO” AND +stops == 0 +ORDER BY airline; +

    Similarly, the following query gathers qualifying index keys from all partitions as + destinationairport IS NOT MISSING is not an equality + predicate.

    +

    Example 7: Create a partitioned index with query non-equality predicate

    + # Lookup all airlines with non-stop flights from SFO +CREATE INDEX idx_pe7 ON `travel-sample` (sourceairport, destinationairport, stops, airline, id) + PARTITION BY HASH (sourceairport, destinationairport); + +SELECT airline, id +FROM `travel-sample` +WHERE sourceairport=”SFO” AND +destinationport is not missing AND +stops == 0 +ORDER BY airline; +

    For the query engine to select qualifying partitions, the partition keys must also be + a part of the index keys. The following index always gathers keys from all + partitions as destinationairport is not an index key.

    +

    Example 8: Create a partitioned index with partition keys not being index + keys

    + # Lookup all airlines with flights from SFO to JFK +CREATE INDEX idx_pe8 ON `travel-sample` (sourceairport, stops, airline, id) + PARTITION BY HASH (sourceairport, destinationairport); + +SELECT airline, id +FROM `travel-sample` +WHERE sourceairport=”SFO” AND +destinationairport=”JFK” +ORDER BY airline; +

    When choosing partition keys other than the document key, the size of each partition + can potentially be subjected to data skew of the chosen partition keys. For example, + for the index in the following example, the partitions containing the major airlines + would have more entries since more index keys would end up hashing into the same + partition.

    + CREATE INDEX idx ON `travel-sample`(airline, destinationairport, sourceairport) + PARTITION BY HASH(airline); +

    During index rebalaning, the rebalancer takes into account the data skew among the + partitions using runtime statistics. It tries to even out resource utilization + across the index service nodes by moving the partitions across the nodes when + possible.

    +
    +
    Choosing Partition Keys for Aggregate Query

    As with a range query, + when an index is partitioned by document key, an aggregate query can gather the + qualifying index keys from all the partitions before performing aggregation in the + query engine. Whenever aggregate pushdown optimization is allowed, the query engine + will push down "partitial aggregate" calculation to each partition. The query engine + then computes the final aggregate result from the partial aggregates across all the + partitions. For more details on aggregate query optimization, see .

    Example + 9: Create a partitioned index with partition key being document + key# Find number of fights out of SFO for every destination across all airlines +CREATE INDEX idx_pe9 ON `travel-sample` (sourceairport, destinationairport, stops, airline, id, ARRAY_COUNT(schedule)) + PARTITION BY HASH (meta().id) where type="route"; + +SELECT sourceairport, destinationairport, SUM(ARRAY_COUNT(schedule)) +FROM `travel-sample` +WHERE sourceairport = "SFO" +AND type = "route" +GROUP BY sourceairport, destinationairport;

    The + choice of partition keys can also improve aggregate query performance when the query + engine can push down the "full aggregate" calculation to the index node. In this + case, the query engine does not have to recompute the final aggregate result from + the index nodes. In addition, certain pushdown optimizations can only be enabled + when a full aggregate result is expected from the index node. To enable a full + aggregate computation, the index must be created with the following requirements: +

      +
    1. The expressions in the GROUP BY clause must match the partition keys.
      2. +
    2. The expressions in the GROUP BY clause must match the leading index + keys.
      3. +
    3. The partition keys must match the leading index keys.
      4. +

    Example 10: Create a partitioned index with the partition keys for full + aggregate pushdown
    + # Find number of fights out of SFO for every destination across all airlines +CREATE INDEX idx_pe10 ON `travel-sample` (sourceairport, destinationairport, stops, airline, id, ARRAY_COUNT(schedule)) + PARTITION BY HASH (sourceairport, destinationairport) where type="route"; + +SELECT sourceairport, destinationairport, SUM(ARRAY_COUNT(schedule)) +FROM `travel-sample` +WHERE sourceairport = "SFO" +AND type = "route" +GROUP BY sourceairport, destinationairport; +
    + Number of Partitions +

    The number of index partitions is fixed when the index is created. By default, each + index will have 8 partitions. The Administrator can override the number of + partitions at index creation time.

    +

    Example 11: Create a partitioned index with 16 partitions

    + CREATE INDEX idx_pe11 ON `travel-sample`(airline, sourceairport, destinationairport) + PARTITION BY HASH(airline) WITH {"num_partition":16}; +
    +
    + Partition Placement +

    When a partitioned index is created, the partitions across are across available index + nodes. During placement of the new index, the index service assumes that each + partition has an equal size and places the partitions according to the availability + of resources on each node. For example, if an index node has more available free + memory than the other nodes, it will assign more partitions to this index node. If + the index has a replica, then the replica partition will not be placed onto the same + node.

    +

    Alternatively, you can specify the node list to restrict the set of nodes available + for placement by using a command similar to the following example.

    +

    Example 12: Create a partitioned index on specific ports of a + nodeCREATE INDEX idx_pe12 ON `travel-sample`(airline, sourceairport, destinationairport) + PARTITION BY KEY(airline) WITH {"nodes":["127.0.0.1:9001", "127.0.0.1:9002"]};

    +

    You can optionally provide sizing hints too. Given the sizing hints, the planner uses + a formula to estimate the memory and CPU usage of the index. Based on the estimated + memory and CPU usage, the planner tries to place the partitions according to the + free resources available to each index node.

    + + Sizing Hints + + + + + + + Optional Sizing Hint + Description + Example + + + + + secKeySize + The average length of the combined index keys + 20 + + + docKeySize + The average length of the document key meta().id + 20 + + + arrSize + The average length of the array field. Non-array fields will be + ignored. + 10 + + + +
    +

    To provide sizing estimation, you can use a command similar to the following + examples.

    +

    Example 13: Create a partitioned index with specific key + sizesCREATE INDEX idx_pe13 ON `travel-sample`(airline, sourceairport, destinationairport) + PARTITION BY HASH (airline) WITH {"secKeySize":20, "docKeySize":20};

    +

    Example 14: Create a partitioned index with specific key and array + sizesCREATE INDEX idx_pe14 ON `travel-sample`(airline, sourceairport, schedule) + PARTITION BY HASH (airline) WITH {"secKeySize":20, "docKeySize":20, "arrSize": 100};

    +
    +
    + Partition Replica +

    A partitioned index can be created with multiple replicas to ensure indexes are + online despite node failure. if there are multiple server groups in a cluster, + replica partitions will be spread out to each server group whenever possible. If one + of the server groups is offline, the remaining replica partitions will be available + to serve all queries. Every index replica is available to serve queries. Therefore, + index replicas can also be used to load rebalancing of query requests.

    +

    Example 15: Create an index with + replicaCREATE INDEX idx_pe15 ON `travel-sample`(airline, sourceairport, schedule) + PARTITION BY HASH (airline) WITH {"num_replica":2};

    +

    When an index node fails, any in-flight query requests (serviced by the failed node) + will fail since the partial results are already being processed. Any new query + requests requiring the lost partition are then serviced by the partitions in the + replica.

    +
    +
    + Rebalancing +

    When new index nodes are added or removed from the cluster, the rebalance operation + attempts to move the index partitions across available index nodes in order to + balance resource consumptions. At the time of rebalancing, the rebalance operation + gathers statistics from each index. These statistics are fed to an optimization + algorithm to determine the possible placement of each partition in order to + minimize the variation of resource consumption across index nodes.

    +

    The rebalancer will only attempt to balance resource consumption on a best try + basis. There are situations where the resource consumption cannot be fully balanced. + For example:

      +
    • The index service will not try to move the index if the cost to move an + index across nodes is too high.
      • +
    • A cluster has a mix of non-partitioned indexes and partitioned indexes.
      • +
    • There is data skew in the partitions.
      • +

    +
    +
    + Repairing Failed Partitions +

    When an index node fails, the index partitions on that node will be lost. The lost + partitions can be recovered or repaired when:

      +
    1. The failed node is delta-recovered.
      2. +
    2. The failed node is rebalanced out of the cluster. The lost partitions on + that node can be repaired/rebuilt in other index nodes whenever possible. + The lost partitions cannot be repaired when the number of remaining nodes is + less than or equal to the number of index replicas.
      3. +

    +
    +
    + Performance Considerations +

    Max_parallelism

    +

    Along with aggregate pushdown optimization, an application can further enhance the + aggregate query performance by computing aggregation in parallel for each partition + in the index service. This can be achieved by specifying the parameter + max_parallelism when issuing a query. The value for + max_parallelism should match the number of partitions of the + index Note than when this is enabled, the index service uses more CPU and memory + since the query traffic is increased according to the value set in the parameter + max_parallelism.

    +

    OFFSET Pushdown

    +

    When there are more than one qualifying partitions involved in a range query, the + query engine will not push down the OFFSET clause to the index service. Without + partition elimination, a partitioned index will have higher overhead for queries + with a large OFFSET value. Alternatively, applications can use + keyset based pagination with partitioned index to achieve good + pagination query performance, detailed in this blog Database Pagination: Using OFFSET and Keyset in + N1QL..

    +

    For aggregate queries, the query engine will pushdown the OFFSET clause whenever full + aggregate result is expected and there is only 1 qualifying partition involved in + the query.

    +

    LIMIT Pushdown

    +

    When there are more than one qualifying partitions involved in a range query, the + query engine will pushdown the LIMIT clause by rewriting it to be the sum of values + in the LIMIT clause and OFFSET clause.

    +

    For aggregate queries, the query engine will pushdown the LIMIT clause whenever a + full aggregate result is expected. When there are more than one qualifying + partitions involved in an aggregate query, the query engine will pushdown the LIMIT + clause by rewriting it to be the sum of values in the LIMIT clause and OFFSET + clause.

    +

    DISTINCT Aggregate Pushdown

    +

    The query engine will not pushdown distinct aggregate calculation to the index node + unless full aggregate result is expected.

    +
    + +     diff --git a/content/n1ql/n1ql-language-reference/insert.dita b/content/n1ql/n1ql-language-reference/insert.dita index f422020f95..6c6975420e 100644 --- a/content/n1ql/n1ql-language-reference/insert.dita +++ b/content/n1ql/n1ql-language-reference/insert.dita @@ -78,7 +78,7 @@ RETURNING META().id as docid, *; bucket. Any user who has the bucket credentials or any Couchbase administrator should be able to insert documents into a bucket. This includes the bucket administrator for the specified bucket, the cluster administrator, and the full administrator roles. See for details about + href="../../security/security-roles.dita"> for details about access privileges for various administrators.

    You cannot insert documents into a SASL bucket if you have a read-only diff --git a/content/n1ql/n1ql-language-reference/let.dita b/content/n1ql/n1ql-language-reference/let.dita index cb350abf93..2762bc8c5a 100644 --- a/content/n1ql/n1ql-language-reference/let.dita +++ b/content/n1ql/n1ql-language-reference/let.dita @@ -1,13 +1,129 @@ - - + + LET clause + Use LET to create variables for later use within a query. - - LET (alias = expression) [, (alias = expression)]* - -

    LET stores the result of a sub-expression in order to use it in subsequent clauses.

    -

    Aliases in the LET clause create new names that can be referred to anywhere in the query.

    - - -     + +

    (Introduced in Couchbase Server 4.0)

    +
    + Purpose +

    In a query expression, it is sometimes useful to store the result of a sub-expression in + order to use it in subsequent clauses. You can do this with the LET + keyword, which creates a new variable and initializes it with the result of the expression + you supply. You can use the LET clause within an array, in a for-loop, or + independently.

    +

    For example, create a variable for the minimum latitude and set it to 71 (detailed in Example + 1):SELECT airportname, geo.lat +FROM `travel-sample` +LET minimum_lat = 71 +WHERE geo.lat > minimum_lat;Without + the LET clause, your complex queries would need to be divided into two + separate queries:

      +
    • One query to get a particular value (or set of values), and
      • +
    • One query to use the value (or values) from the first query.
      • +
    If the LET variable is referenced in the WHERE + clause, then it is evaluated before the WHERE clause; otherwise, it is + evaluated after the WHERE clause.

    +

    The LET clause can't be chained and can't be referenced by another + LET variable.

    +

    Each LET alias needs to be unique within its scope.

    +
    +
    + Prerequisites +

    The LET clause can only be used in a SELECT statement, + and in order for you to select data from a document or keyspace, you must have the + query_select privilege on the document or keyspace. For more details + about user roles, see .

    +
    +
    + Syntax +

    +

    + LET (alias = expr) [, (alias2 = expr2)]* +

    +
    +
    + Arguments +

    +

    + +
    alias
    +
    [Required] String or expression that represents the name of the variable.
    +     +
    +
    + +
    expr
    +
    [Required] String or expression that represents the value assigned to its + alias.
    +     +
    +

    +
    +
    + Examples +

    Example 1: Use LET to set variables to a number, an operation on a + field, and a subquery.

    +

    Find all airports and cities between certain latitudes in a country with a landmark.

    +

    SELECT t1.airportname, t1.geo.lat, t1.geo.lon, t1.city, t1.type +FROM `travel-sample` t1 +LET min_lat = 71, max_lat = ABS(t1.geo.lon)*4+1, + place = (SELECT RAW t2.country + FROM `travel-sample` t2 + WHERE t2.type = "landmark") +WHERE t1.type = "airport" +AND t1.geo.lat > min_lat +AND t1.geo.lat < max_lat +AND t1.country IN place;Results:[ + { + "airportname": "Wiley Post Will Rogers Mem", + "city": "Barrow", + "lat": 71.285446, + "type": "airport" + }, + { + "airportname": "Dillant Hopkins Airport", + "city": "Keene", + "lat": 72.270833, + "type": "airport" + } +] +

    +

    Example 2: Use LET to set a variable in an array.

    +

    Find all Sunday flights (day = 0) to the Charles De Gaulle airport + (CDG) on Air India (AI) airlines. In this example, + the LET variable sch is not used in the WHERE clause but + used only in the projection. Therefore, the Query Planner defers the evaluation to + post-predicate evaluation, so there is no overhead for documents that are not qualified by + the predicates.

    +

    SELECT t1.airline, t1.destinationairport, sch AS schedule +FROM `travel-sample` AS t1 +LET sch = ARRAY v FOR v IN t1.schedule WHEN v.day = 0 END +WHERE type = "route" +AND t1.destinationairport = "CDG" +AND t1.airline = "AI"; +Results:[ + { + "airline": "AI", + "destinationairport": "CDG", + "schedule": [ + { + "day": 0, + "flight": "AI988", + "utc": "00:24:00" + }, + { + "day": 0, + "flight": "AI972", + "utc": "17:32:00" + } + ] + } +] +

    +
    + + diff --git a/content/n1ql/n1ql-language-reference/limit.dita b/content/n1ql/n1ql-language-reference/limit.dita index 20da0d152c..ebd0e3df4a 100644 --- a/content/n1ql/n1ql-language-reference/limit.dita +++ b/content/n1ql/n1ql-language-reference/limit.dita @@ -1,21 +1,47 @@ - - + + LIMIT clause + The LIMIT clause specifies the maximum number of documents to be + returned in a resultset by a SELECT statement. - -LIMIT expression -

    The LIMIT clause specifies the maximum number of objects that can be returned in a result set - by SELECT. A negative value or a value greater than 9223372036854775295 (result of - math.MaxInt64- 512) is considered as LIMIT 0. Starting from - version 4.5, the LIMIT clause in DML statements is no longer a hint. It indicates that the - actual number of mutations will be less than or equal to the specified LIMIT.

    - SELECT t.name, t.address, t.city, t.country, t.url -FROM `travel-sample` t + +

    (Introduced in Couchbase Server 4.0)

    +
    + Purpose +

    When you don't need the entire resultset, use the LIMIT clause to specify + the maximum number of documents to be returned in a resultset by a SELECT + query.

    +

    The LIMIT and OFFSET clauses are evaluated after the ORDER BY clause.

    +

    A negative value is the same as LIMIT 0. Starting from version 4.5, + the LIMIT clause in INSERT, UPDATE, and DELETE statements is no longer a + hint. It indicates that the actual number of mutations will be less than or equal to the + specified LIMIT.

    +
    +
    + Syntax +

    LIMIT expr

    +
    +
    + Arguments +

    +

    + +
    expr
    +
    Integer or an expression that evaluates to an integer representing the number of + resulting documents.
    +     +
    +

    +
    +
    + Examples +

    Example 1: Get only 2 documents of hotels with an empty room.SELECT name, address, city, country, url +FROM `travel-sample` WHERE type="hotel" - AND t.vacancy = true -LIMIT 2; - [ + AND vacancy = true +LIMIT 2; Result:[ { "address": "Capstone Road, ME7 3JE", "city": "Medway", @@ -30,7 +56,28 @@ WHERE type="hotel" "name": "The Robins", "url": "http://givernyguesthouse.com/robin.htm" } -] - - - +] + Example 2: Set the limit of Example 1 based on an + equation.SELECT name, address, city, country, url +FROM `travel-sample` +WHERE type="hotel" + AND vacancy = true +LIMIT (20/10);Result:[ + { + "address": "Capstone Road, ME7 3JE", + "city": "Medway", + "country": "United Kingdom", + "name": "Medway Youth Hostel", + "url": "http://www.yha.org.uk" + }, + { + "address": "6 rue aux Juifs", + "city": "Giverny", + "country": "France", + "name": "The Robins", + "url": "http://givernyguesthouse.com/robin.htm" + } +]

    +
    + +     diff --git a/content/n1ql/n1ql-language-reference/n1ql-auditing.dita b/content/n1ql/n1ql-language-reference/n1ql-auditing.dita new file mode 100644 index 0000000000..e9a16e93c4 --- /dev/null +++ b/content/n1ql/n1ql-language-reference/n1ql-auditing.dita @@ -0,0 +1,726 @@ + + + + + N1QL Auditing + + + N1QL-related activities can be audited, by Couchbase Server. + + + + +
    + + + Understanding N1QL Auditing + + +

    + This section provides specific information on Couchbase Server auditing + as it relates to N1QL. For a general description of configuring auditing + with Couchbase Web Console, see + Auditing. +

    + +

    + Couchbase Server provides auditing for N1QL-related + activities such as the following: +

    + +
      +
    • Authenticating
      • +
    • Starting and stopping the Query Service
      • +
    • Editing Query Service settings
      • +
    • Executing N1QL statements
      • +
    • Non-query API requests
      • +
    + +

    + N1QL-related activities are logged whether they are executed + by a person or by an application running on behalf of a person. Auditing + occurs at the level of requests, rather than of operations. Thus, + when a + request arrives with a SELECT query, only the SELECT query itself is logged: the + associated subsidiary operations performed by the Data and Index Services are + not logged. +

    + +

    + Auditing causes a reduction in N1QL query-performance. This is in the range of + 9% to 17% of queries performed per second: the exact reduction depends + on query-size, and on the amount of auditing that has been enabled. Large queries + and minimal auditing cause less performance-reduction. +

    + +

    + Auditing can be configured by means of Couchbase Web Console: see the information provided + in + Auditing. + To capture N1QL-related events, use the Query and Index Service panel. Events available + to be audited include ones issued through the SDK, the Query workbench, and the Query Shell. +

    + + +
    + +
    Audit Log Format

    The audit records are + written in JSON format to match the format used for Admin Auditing to allow easy + integration with downstream auditing tools for audit log analysis. The syslog format + will allow for integration with third party SIEM tools, such as + QRadar.

    Required auditing fields for executed statements: + + + + + + + Field + Description + Example + + + + + timestamp + Exact date and time of the access event in UTC + format. + 2018-02-09T14:52:35.163-08:00 + + + real_userid + Source/User from basic authentication fields of + request. + "source":"local",

    "user":"Administrator"

     +     + + requestId + UUID of request, generated by the N1QL server. + aee53bf0-d009-4015-8a1d-efec74f2cd74 + + + statement + The actual N1QL query that was executed. + SELECT * FROM `travel-sample` + + + isAdHoc + TRUE for statements made + directly.

    FALSE for prepared + statements.

     + TRUE +     + + userAgent + To identify the type of user by a combination of the + User-Agent and CB-User-Agent headers in one of the four + formats:

    1) CURL request

    2) Query + Workbench

    3) CBQ shell

    4) + SDK

     + 1) curl/7.43.0

    2) Mozilla/5.0 + (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 + (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36 + (Couchbase Query Workbench + (5.1.0-1434-enterprise))

    3) + Go-http-client/1.1 (CBQ/2.0)

    4) + couchbase-java-client/2.5.2 (git: 2.5.2, core: + 1.5.2) (Mac OS X/10.11.6 x86_64; Java HotSpot(TM) 64-Bit + Server VM 1.8.0_101-b13)

     +     + + node + Assigned name (IP address) of the server where the request + ran.

    local for unclustered + nodes.

     + local +     + + status + Status of the request, as success or + failed or stopped. + success + + + metrics + The elapsed time (ms), execution time (ms), result count, and + result size (MB). + "elapsedTime":"7.599684ms",

    "executionTime":"7.507755ms",

    "resultCount":0,

    "resultSize":0

     +     + + id + Number for the audit event type. + 28672 + + + name + The N1QL command or REST API request type. + SELECT + + + description + Description of the event type. + A N1QL SELECT statement was executed + + + +

    Optional auditing fields for statements: + + + + + + + Field + Description + Example + + + + + namedArgs + Names and values of name arguments. + $val and $user + + + positionalArgs + Array of values of positional arguments. + $1 and ? + + + clientContextId + Captured from the client_context_id parameter of + the N1QL query API.

    May be used to distinguish between + user-generated queries and UI-generated queries from the Query + WorkBench.

    UI-generated queries have the prefix + INTERNAL- in this field. The client + context ID has no security guarantees. The parameter can be set + by any user in any request and is not verified in the server, so + it should not be relied upon for security + purposes.     + +     + + +

    Required auditing fields for API requests: + + + + + + + Field + Description + Example + + + + + timestamp + Exact date and time of the access event in UTC + format. + 2018-02-09T14:52:35.163-08:00 + + + real_userid + Source/User from basic authentication fields of + request. + "source":"local","user":"Administrator" + + + httpMethod + The API method call, either GET, PUT, DELETE, + POST + GET + + + httpResultCode + The number representing the API result. + 200 + + + errorMessage + If an error occurred, this will contain information on the + error. + User does not have credentials to run queries + accessing the system tables. + + + id + Number for the API + auditing code. + 28689 + + + name + The API request location. + /admin/ping + + + description + Description of the event type. + An HTTP request was made to the API at + /admin/ping. + + + +

    +
    Examples

    To reduce disk usage and improve + performance, the log files are as compact as possible.

    When viewed through + Query Workbench, the logs are formatted and indented for easier + reading.

    Example 1: Execute SELECT * FROM orders + via a CURL + statement.

    {"timestamp":"2018-02-09T14:52:35.163-08:00","real_userid":{"source":"local","user":"Administrator"},"requestId":"aee53bf0-d009-4015-8a1d-efec74f2cd74","statement":"SELECT * FROM orders","isAdHoc":true,"userAgent":"curl/7.43.0","node":"local_node","status":"success","metrics":{"elapsedTime":"7.599684ms","executionTime":"7.507755ms","resultCount":0,"resultSize":0},"id":28672,"name":"SELECT statement","description":"A N1QL SELECT statement was executed”}To + make the log entry easier-to-read:

      +
    1. Copy the log entry to the clipboard
      2. +
    2. Open Query Workbench
      3. +
    3. Type SELECT
      4. +
    4. Paste the query from the clipboard
      5. +
    5. Click the Execute button
      6. +
    [ + { + "$1": { + "description": "A N1QL SELECT statement was executed", + "id": 28672, + "isAdHoc": true, + "metrics": { + "elapsedTime": "7.599684ms", + "executionTime": "7.507755ms", + "resultCount": 0, + "resultSize": 0 + }, + "name": "SELECT statement", + "node": "local_node", + "real_userid": { + "source": "local", + "user": "Administrator" + }, + "requestId": "aee53bf0-d009-4015-8a1d-efec74f2cd74", + "statement": "SELECT * FROM orders", + "status": "success", + "timestamp": "2018-02-09T14:52:35.163-08:00", + "userAgent": "curl/7.43.0" + } + } +] +

    Example + 2: Execute DELETE FROM orders WHERE priority = 6 via a CURL statement

    + {"timestamp":"2018-02-09T14:52:55.786-08:00","real_userid":{"source":"local","user":"Administrator"},"requestId":"ded68ae3-d964-4d87-b1c2-70cf72041c6b","statement":"DELETE FROM orders WHERE priority = 6","isAdHoc":true,"userAgent":"curl/7.43.0","node":"local_node","status":"success","metrics":{"elapsedTime":"8.884558ms","executionTime":"8.853976ms","resultCount":0,"resultSize":0},"id":28678,"name":"DELETE statement","description":"A N1QL DELETE statement was executed"} +

    [ + { + "$1": { + "description": "A N1QL DELETE statement was executed", + "id": 28678, + "isAdHoc": true, + "metrics": { + "elapsedTime": "8.884558ms", + "executionTime": "8.853976ms", + "resultCount": 0, + "resultSize": 0 + }, + "name": "DELETE statement", + "node": "local_node", + "real_userid": { + "source": "local", + "user": "Administrator" + }, + "requestId": "ded68ae3-d964-4d87-b1c2-70cf72041c6b", + "statement": "DELETE FROM orders WHERE priority = 6", + "status": "success", + "timestamp": "2018-02-09T14:52:55.786-08:00", + "userAgent": "curl/7.43.0" + } + } +] +

    Example + 3: Make an HTTP GET method from an + /admin/ping API + request.{"timestamp":"2018-02-09T14:53:10.856-08:00","real_userid":{"source":"internal","user":"unknown"},"httpMethod":"GET","httpResultCode":200,"errorMessage":"","id":28697,"name":"/admin/ping API request","description":"An HTTP request was made to the API at /admin/ping."}[ + { + "$1": { + "description": "An HTTP request was made to the API at /admin/ping.", + "errorMessage": "", + "httpMethod": "GET", + "httpResultCode": 200, + "id": 28697, + "name": "/admin/ping API request", + "real_userid": { + "source": "internal", + "user": "unknown" + }, + "timestamp": "2018-02-09T14:53:10.856-08:00" + } + } +] +

    + +
    + Audit Rotation +

    The auditing Rotation parameters can be only one of the following:

    +

    + + + + + + + Audit Log Rotation Type + Examples + + + + + Time-based (days) + 7 (for weekly); 30 (for monthly). + + + Size-based (MB) + 10 (for 10 MB); 10000 (for 10 GB). + + + +
    +

    +
    +
    + Audit Failure Semantics +

    When the audit target fails, the auditing system can be set to one of the + following: + + + + + + Failure Response Type + Description + + + + + Ignore + Continue the action without firing an audit record. + + + Block + Cancel the operation. + + + Log Reuse + This option is for out-of-space failures:
      +
    • Time-Based: Limit audit logs to the specified + number of recent days.
      • +
    • Size-Based: Limit audit log size to the specified + number of megabytes.
      • +
    +     + + +
    If an audit record attempt fails in the query engine, an error message will + be printed to the query.log file.

    +
    +
    + Audit Trail Protection +

    To prevent unauthorized modification of the audit service configuration, the auditing + system restricts access to configuring only to Full and Security Administrators.

    +

    Audit records are immutable since the auditing system prevents changes of audit event + records once written.

    +

    Once archived, audit data is deleted from Couchbase, and the file space is recovered.

    +

    + The + cbcollect_info + utility does not collect audit logs. +

    +
    +
    + Audit Event Types +

    Below is the list of all events that are captured in the audit logs.

      +
    1. System clock modifications as captured in the operating system audit + log
      2. +
    2. Disabling Auditing
      3. +
    3. Enabling Auditing, with audit settings written
      4. +
    4. Login, both Success and Failure
      5. +
    5. Logout, both Success and Failure
      6. +
    6. Data access operations + + + + + + a. Query Service + Event ID + + + + + ALTER INDEX + 28683 + + + BUILD INDEX + 28684 + + + CREATE INDEX + 28681 + + + CREATE PRIMARY INDEX + 28688 + + + DELETE + 28678 + + + DROP INDEX + 28682 + + + EXPLAIN + 28673 + + + GRANT + 28685 + + + INFER + 28675 + + + INSERT + 28676 + + + MERGE + 28680 + + + PREPARE + 28674 + + + REVOKE + 28686 + + + SELECT + 28672 + + + UNRECOGNIZED + 28687 + + + UPDATE + + 28679 + + + UPSERT + + 28677 + + + +
      + + + + + + b. API Request + Event ID + + + + + /admin/active_requests + 28692 + + + /admin/clusters + 28701 + + + /admin/completed_requests + 28702 + + + /admin/config + 28698 + + + /admin/indexes/active_requests + 28694 + + + /admin/indexes/completed_requests + 28702 + + + /admin/indexes/prepareds + 28693 + + + /admin/ping + 28697 + + + /admin/prepareds + 28691 + + + /admin/settings + 28700 + + + /admin/ssl_cert + 28699 + + + /admin/stats + 28689 + + + /admin/vitals + 28690 + + + +
      7. +
    7. Audit-Archive
      8. +
    8. System-Backup
      9. +
    9. Data service
        +
      1. Read
        2. +
      2. Write
        3. +
      3. DCP-Read
        4. +
      4. DCP-Write
        5. +
      10. +
    10. FTS Service
        +
      1. FTS-Read
        2. +
      11. +
    11. Analytics audit events

      Items that will not be captured in the audit + logs:

        +
      • API calls that are not statements
        • +
      • API requests sent to URLs the query engine does not service
        • +
      • API requests which are handled by the autonomic functionality of + the HTTP server
        • +

      12. +

    +
    +
    + API Auditing Codes +

    Audit records will be issued by the query engine for requests to its secondary APIs. + This does not include the main URL used for queries (/query/service) but does + include all other URLs the query engine listens to.

    +

    There will be a separate audit record code for each registered URL. The mapping from + URLs to audit record codes is given below. Some URLs require extra fields, as + noted. + + + + + + + Audit Code + API + Remarks + + + + + 28689 + /admin/stats +

    /admin/stats/{stat}

     + Field "stat": optional, string, for input parameter {stat} if + present. +     + + 28690 + /admin/vitals + + + + 28691 + /admin/prepareds +

    /admin/prepareds/{name}

     + Field "name": optional, string, for input parameter {name} if + present.

    Do not audit POST requests.

     +     + + 28692 + /admin/active_requests +

    /admin/active_requests/{request}

     + Field "request": optional, string, for input parameter + {request} if present.

    Do not audit POST requests.

     +     + + 28693 + /admin/indexes/prepareds + + + + 28694 + /admin/indexes/active_requests + + + + 28695 + /admin/indexes/completed_requests + + + + 28696 + /debug/vars + + + + 28697 + /admin/ping + + + + 28698 + /admin/config + + + + 28699 + /admin/ssl_cert + + + + 28700 + /admin/settings + + + + 28701 + /admin/clusters

    /admin/clusters/{cluster}

    /admin/clusters/{cluster}/nodes

    /admin/clusters/{cluster}/nodes/{node}

     + Field "cluster": optional, string, for input parameter + {cluster} if present.

    Field "node": optional, string, for + input parameter {node} if present.

    Field "body": + PUT/POST only, JSON representation of cluster or node from + request body.

     +     + + 28702 + /admin/completed_requests

    /admin/completed_requests/{request}

     + Field "request": optional, string, for input parameter + {request} if present.

    Do not audit POST requests.

     +     + + +

    +
    + +     diff --git a/content/n1ql/n1ql-language-reference/n1ql-langref.ditamap b/content/n1ql/n1ql-language-reference/n1ql-langref.ditamap index e63faab58f..e91e754cdc 100644 --- a/content/n1ql/n1ql-language-reference/n1ql-langref.ditamap +++ b/content/n1ql/n1ql-language-reference/n1ql-langref.ditamap @@ -7,6 +7,7 @@ + @@ -15,6 +16,7 @@ + @@ -53,12 +55,14 @@ + + diff --git a/content/n1ql/n1ql-language-reference/offset.dita b/content/n1ql/n1ql-language-reference/offset.dita index e50799a777..bb5394b1f2 100644 --- a/content/n1ql/n1ql-language-reference/offset.dita +++ b/content/n1ql/n1ql-language-reference/offset.dita @@ -1,48 +1,58 @@ - - + + OFFSET clause + The OFFSET clause specifies the number of resultset objects to skip in + a SELECT query. - -OFFSET expression - -

    The OFFSET clause specifies a number of objects to be skipped. If a LIMIT clause is also present, the OFFSET is - applied prior to the LIMIT. The OFFSET value must be a non-negative integer.

    -

    The OFFSET clause optionally follows the LIMIT clause. If an offset is specified, the - specified number of objects is omitted from the result set before enforcing a specified LIMIT. - This clause must be a non-negative integer.

    - -

    Examples

    -

    The following examples show the LIMIT and OFFSET clauses.

    - - -

    Name and age list: limit and offset by 2

    - SELECT fname, age - FROM tutorial - ORDER BY age - LIMIT 2 OFFSET 2 - -

    Returns:

    - -{ - "results": [ - { - "age": 40, - "fname": "Jane" - }, - { - "age": 46, - "fname": "Dave" - } - ] - } - -

    Golf products list: limit and offset by 10

    - SELECT * - FROM product - UNNEST product.categories AS cat - WHERE LOWER(cat) IN ["golf"] LIMIT 10 OFFSET 10 - - -     -     + +

    (Introduced in Couchbase Server 4.0)

    +
    + Purpose +

    When you want the resultset to skip over the first few resulting objects, use the + OFFSET clause to specify that number of objects to ignore.

    +

    The LIMIT and OFFSET clauses are evaluated after the ORDER BY clause.

    +

    If a LIMIT clause is also present, the OFFSET is applied + prior to the LIMIT; that is, the specified number of objects is omitted + from the result set before enforcing a specified LIMIT.

    +
    +
    Syntax

    + OFFSET expr +

    +
    + Arguments +

    +

    + +
    expr
    +
    Integer or an expression that evaluates to an integer which is non-negative.
    +     +
    +

    +
    +
    + Examples +

    Example 1: List 4 airport cities after skipping the first + 200.SELECT DISTINCT city +FROM `travel-sample` +WHERE type = "airport" +ORDER BY city +LIMIT 4 +OFFSET 200;Results:[ + { + "city": "Brownsville" + }, + { + "city": "Brownwood" + }, + { + "city": "Brunswick" + }, + { + "city": "Bryan" + } +]

    +
    +

    + + diff --git a/content/n1ql/n1ql-language-reference/orderby.dita b/content/n1ql/n1ql-language-reference/orderby.dita index f6c7bb3256..5e596bcb52 100644 --- a/content/n1ql/n1ql-language-reference/orderby.dita +++ b/content/n1ql/n1ql-language-reference/orderby.dita @@ -1,65 +1,143 @@ - - + + ORDER BY clause + The ORDER BY clause sorts the result-set by one or more columns, in + ascending or descending order. - -

    order-by-clause:

    -ORDER BY ordering-term [, ordering-term]* -

    ordering-term:

    - expression [ASC | DESC] - -

    The ORDER BY clause lets your order the result set based on an expression. The order of items in the result - set is determined by the expression specified in this clause. Objects are sorted first by the left-most expression - in the list of expressions. Any items with the same sort value will be sorted with the next expression in the list. - This process repeats until all items are sorted and all expressions in the list are evaluated. You can specify whether - to order the results in ascending or descending order.

    - -

    If no ORDER BY clause is specified, the order in which the result objects are returned is undefined.

    -

    As ORDER BY expressions can evaluate to any JSON value, we define an ordering when comparing values of different types. - The following list describes the order by type (from lowest to highest):

    -
      -
    • missing value
      • -
    • null
      • -
    • false
      • -
    • true
      • -
    • number
      • -
    • string (string comparison is done using a raw byte collation of UTF8 encoded strings)
      • -
    • array (element by element comparison is performed until the end of the shorter array; if all the elements - so far are equal, then the longer array sorts after)
      • -
    • object (larger objects sort after; for objects of equal length, key/value by key/value comparison is performed; keys - are examined in sorted order using the normal ordering for strings)
      • -
    - -

    The following example shows the ORDER BY clause.

    -SELECT fname, age - FROM tutorial - ORDER BY age - - -

    Returns:

    - - { - "results": [ - { - "age": 18, - "fname": "Fred" - }, - { - "age": 20, - "fname": "Harry" - }, - { - "age": 40, - "fname": "Jane" - }, - { - "age": 46, - "fname": "Dave" - } - ] - } - - - -     + +
    + Purpose +

    In a SELECT statement, the ORDER BY clause sorts the + result-set in ascending or descending order, based on one or more fields or expressions of + those fields in the projection.

    +
    +
    + Prerequisites +

    For you to select data from a document or keyspace, you must have the + query_select privilege on the document or keyspace. For more details + about user roles, see .

    +
    +
    SyntaxIn the below SELECT + statement, the ORDER BY clause is boldfaced:

    + SELECT select_clause +[FROM from_clause] [JOIN join_clause] +[USE INDEX useindex_clause] +[LET let_clause] +[WHERE where_clause ( [AND where_clause2] )* ] +[GROUP BY groupby_clause] [LETTING|HAVING letting_clause] +[UNION|INTERSECT|EXCEPT union_clause] +[ORDER BY orderby_clause] +[LIMIT limit_int] +[OFFSET offset_clause] +; +

    Syntax of the + orderby_clauseORDER BY expr [ASC|DESC] [, expr2 [ASC|DESC] ]*
    +
    + Arguments +

    +

    + +
    expr
    +
    [Required] This identifier or expression can be document fields, new expressions, or + an alias in the SELECT clause.
    +     +
    +
    + +
    ASC | DESC
    +
    [Optional; default is ASC]
    +
    ASC orders in ascending order.
    +
    DESC orders in descending order.
    +     +
    +

    +
    +
    + Return Values +

    If no ORDER BY clause is specified, the order in which the result objects + are returned is undefined.

    +

    Objects are sorted first by the left-most expression in the list of expressions. Any items + with the same sort value will be sorted with the next expression in the list. This process + repeats until all items are sorted and all expressions in the list are evaluated.

    +

    When a field has a mix of data types, the different JSON types are sorted in the following + order (from lowest to highest):

      +
    • MISSING
      • +
    • NULL (including JSON NULL)
      • +
    • FALSE
      • +
    • TRUE
      • +
    • number
      • +
    • string (string comparison is done using a raw byte collation of UTF8 encoded + strings)
      • +
    • array (element by element comparison is performed until the end of the shorter array; + if all the elements so far are equal, then longer arrays sort after)
      • +
    • object (larger objects sort after; for objects of equal length, key/value by key/value + comparison is performed; keys are examined in sorted order using the normal ordering for + strings)
      • +

    + Among string values, the ascending order is lowercase, then uppercase, then accented + letters. +
    +
    + Examples +
      +
    • Ex 1: List cities in descending order + and then landmarks in ascending order.
      • +
    • Ex 2: List names resulting from a UNION + clause.
      • +
    +

    Example 1: List cities in descending order and then landmarks in ascending + order.SELECT city, name +FROM `travel-sample` +WHERE type = "landmark" +ORDER BY city DESC, name ASC;Results:[ + { + "city": "Évreux", + "name": "Cafe des Arts" + }, + { + "city": "Épinal", + "name": "Marché Couvert (covered market)" + }, + { + "city": "Épinal", + "name": "Musée de l'Image/Imagerie d'Épinal" + }, + { + "city": "Yosemite Valley", + "name": "Lower Yosemite Fall" + }, + { + "city": "Yosemite Valley", + "name": "Mirror Lake/Meadow" + }, +...

    +
    +

    Example 2: List the names of hotels and landmarks resulting from a UNION + query.SELECT name + FROM `travel-sample` + WHERE type = "landmark" +UNION SELECT name + FROM `travel-sample` + WHERE type = "hotel" +ORDER BY name ASC;Results:{ + "name": "'La Mirande Hotel" + }, + { + "name": "'The Argyll Arms Hotel" + }, + { + "name": "'Visit the Hut of the Shadows and other End of the Road sculptures" + }, + { + "name": "02 Shepherd's Bush Empire" + }, + { + "name": "101 Coffee Shop" + }, +...

    +

    + + diff --git a/content/n1ql/n1ql-language-reference/reservedwords.dita b/content/n1ql/n1ql-language-reference/reservedwords.dita index 071ea81c69..a158025157 100644 --- a/content/n1ql/n1ql-language-reference/reservedwords.dita +++ b/content/n1ql/n1ql-language-reference/reservedwords.dita @@ -46,249 +46,254 @@ ANY ARRAY - - AS - ASC - BEGIN - BETWEEN - BINARY - BOOLEAN - - - BREAK - BUCKET - BUILD - BY - CALL - CASE - - - CAST - CLUSTER - COLLATE - COLLECTION - COMMIT - CONNECT - - - CONTINUE - CORRELATE - COVER - CREATE - DATABASE - DATASET - - - - DATASTORE - DECLARE - DECREMENT - DELETE - DERIVED - DESC - - - - DESCRIBE - DISTINCT - DO - DROP - EACH - ELEMENT - - - - ELSE - END - EVERY - EXCEPT - EXCLUDE - EXECUTE - - - - EXISTS - EXPLAIN - FALSE - FETCH - FIRST - FLATTEN - - - FOR - FORCE - FROM - FUNCTION - GRANT - GROUP - - - - GSI - HAVING - IF - IGNORE - ILIKE - IN - - - - INCLUDE - INCREMENT - INDEX - INFER - INLINE - INNER - - - - INSERT - INTERSECT - INTO - IS - JOIN - KEY - - - KEYS - KEYSPACE - KNOWN - LAST - LEFT - LET - - - - LETTING - LIKE - LIMIT - LSM - MAP - MAPPING - - - - MATCHED - MATERIALIZED - MERGE - MINUS - MISSING - NAMESPACE - - - - NEST - NOT - NULL - NUMBER - OBJECT - OFFSET - - - - ON - OPTION - OR - ORDER - OUTER - OVER - - - - PARSE - PARTITION - PASSWORD - PATH - POOL - PREPARE - - - - PRIMARY - PRIVATE - PRIVILEGE - PROCEDURE - PUBLIC - RAW - - - - REALM - REDUCE - RENAME - RETURN - RETURNING - REVOKE - - - - RIGHT - ROLE - ROLLBACK - SATISFIES - SCHEMA - SELECT - - - - SELF - SEMI - SET - SHOW - SOME - START - + + AS + ASC + BEGIN + BETWEEN + BINARY + BOOLEAN + + + BREAK + BUCKET + BUILD + BY + CALL + CASE + + + CAST + CLUSTER + COLLATE + COLLECTION + COMMIT + CONNECT + + + CONTINUE + CORRELATE + COVER + CREATE + DATABASE + DATASET + + + + DATASTORE + DECLARE + DECREMENT + DELETE + DERIVED + DESC + - - STATISTICS - STRING - SYSTEM - THEN - TO - TRANSACTION - + + DESCRIBE + DISTINCT + DO + DROP + EACH + ELEMENT + + + + ELSE + END + EVERY + EXCEPT + EXCLUDE + EXECUTE + + + + EXISTS + EXPLAIN + FALSE + FETCH + FIRST + FLATTEN + + + FOR + FORCE + FROM + FUNCTION + GRANT + GROUP + + + + GSI + HAVING + IF + IGNORE + ILIKE + IN + + + + INCLUDE + INCREMENT + INDEX + INFER + INLINE + INNER + + + + INSERT + INTERSECT + INTO + IS + JOIN + KEY + + + KEYS + KEYSPACE + KNOWN + LAST + LEFT + LET + + + + LETTING + LIKE + LIMIT + LSM + MAP + MAPPING + - - TRIGGER - TRUE - TRUNCATE - UNDER - UNION - UNIQUE - + + MATCHED + MATERIALIZED + MERGE + MINUS + MISSING + NAMESPACE + + + + NEST + NOT + NULL + NUMBER + OBJECT + OFFSET + + + + ON + OPTION + OR + ORDER + OUTER + OVER + + + + PARSE + PARTITION + PASSWORD + PATH + POOL + PREPARE + + + + PRIMARY + PRIVATE + PRIVILEGE + PROCEDURE + PUBLIC + RAW + - - UNKNOWN - UNNEST - UNSET - UPDATE - UPSERT - USE - + + REALM + REDUCE + RENAME + RETURN + RETURNING + REVOKE + + + + RIGHT + ROLE + ROLLBACK + SATISFIES + SCHEMA + SELECT + + + + SELF + SEMI + SET + SHOW + SOME + START + + + + STATISTICS + STRING + SYSTEM + THEN + TO + TRANSACTION + - - USER - USING - VALIDATE - VALUE - VALUED - VALUES - - - - VIA - VIEW - WHEN - WHERE - WHILE - WITH - + + TRIGGER + TRUE + TRUNCATE + UNDER + UNION + UNIQUE + + + + UNKNOWN + UNNEST + UNSET + UPDATE + UPSERT + USE + + + + USER + USING + VALIDATE + VALUE + VALUED + VALUES + + + + VIA + VIEW + WHEN + WHERE + WHILE + WITH + - - WITHIN - WORK - XOR - + + WITHIN + WORK + XOR + + + + + +

    diff --git a/content/n1ql/n1ql-language-reference/selectclause.dita b/content/n1ql/n1ql-language-reference/selectclause.dita index 65dca1a545..f91c90d712 100644 --- a/content/n1ql/n1ql-language-reference/selectclause.dita +++ b/content/n1ql/n1ql-language-reference/selectclause.dita @@ -2,10 +2,12 @@ SELECT Clause + The SELECT clause determines the result set.
    Purpose -

    In a SELECT statement, the SELECT clause determines the result set.

    +

    In a SELECT statement, the SELECT clause determines the + projection (result set).

    Prerequisites @@ -14,25 +16,27 @@ about user roles, see .

    -
    SyntaxIn the below SELECT statement, the SELECT - clause is boldfaced:

    +

    SyntaxIn the below SELECT + statement, the SELECT clause is boldfaced:

    SELECT select_clause -[FROM from_clause] -[JOIN join_clause] +[FROM from_clause] [JOIN join_clause] +[USE INDEX useindex_clause] +[LET let_clause] [WHERE where_clause ( [AND where_clause2] )* ] -[GROUP BY group_clause] -[HAVING having_clause] +[GROUP BY groupby_clause] [LETTING|HAVING letting_clause] [UNION|INTERSECT|EXCEPT union_clause] -[ORDER BY order_clause ] +[ORDER BY orderby_clause] [LIMIT limit_int] [OFFSET offset_clause] ; -

    select_clause::= SELECT [ALL|DISTINCT] ( result-expr [ , result-expr ]* | - (RAW|ELEMENT|VALUE) field_expr [ [AS] alias_name ] )

    Railroad Diagrams - for SELECT clause:

    result-expr:

    path:

    alias:

    Syntax of the + select_clauseSELECT [ALL|DISTINCT] ( result-expr [ , result-expr2 ]* +| (RAW|ELEMENT|VALUE) expr [ [AS] alias ] )

    Railroad + Diagrams for SELECT clause:

    result-expr:

    path:

    alias:

    Arguments @@ -144,7 +148,11 @@ Results:
    result-expr
    One or more expressions that evaluate to one or more field names to be in the - ResultSet. For details with examples, see +
    If no field name is specified, the input for the query is a single empty object that + allows you to perform calculations with the SELECT statement, such as + SELECT 10+20 AS Total; or other N1QL expression.
    +
    For details with examples, see N1QL Expressions.
     diff --git a/content/n1ql/n1ql-language-reference/selectintro.dita b/content/n1ql/n1ql-language-reference/selectintro.dita index 4baf8db115..7f2e6524b8 100644 --- a/content/n1ql/n1ql-language-reference/selectintro.dita +++ b/content/n1ql/n1ql-language-reference/selectintro.dita @@ -310,10 +310,11 @@ LIMIT 2;[ "airlines".

    This example illustrates a special kind of JOIN where the documents on the right side of JOIN contain the foreign key reference to the documents on the left - side. Such JOINs are referred to as index JOIN. See for more details.

    Index JOIN - requires a special inverse index route_airlineid on the JOIN key - ‘route.airlineid’. Create this index using the following command: + side. Such JOINs are referred to as index JOIN. For details, see Lookup JOIN + Clause.

    Index JOIN requires a special inverse index + route_airlineid on the JOIN key ‘route.airlineid’. Create this + index using the following command: CREATE INDEX route_airlineid ON `travel-sample`(airlineid) WHERE type = "route";

    Now we can execute the following query. diff --git a/content/n1ql/n1ql-language-reference/stringfun.dita b/content/n1ql/n1ql-language-reference/stringfun.dita index 0ac559c2cf..584f3c1fda 100644 --- a/content/n1ql/n1ql-language-reference/stringfun.dita +++ b/content/n1ql/n1ql-language-reference/stringfun.dita @@ -12,7 +12,7 @@ (e.g. an integer instead of a string), then NULL is returned as the result.

    - CONTAINS(input_string, search_string) + CONTAINS(in_str, search_str)
    Description
    @@ -25,12 +25,12 @@
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to search within.
     -
    search_string
    +
    search_str
    A string, or any valid expression which evaluates to a string, that is the string to search for.
     @@ -60,7 +60,7 @@
    - INITCAP(input_string) + INITCAP(in_str)
    Description
    @@ -72,7 +72,7 @@
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to convert to title case.
     @@ -106,7 +106,7 @@
    - LENGTH(input_string) + LENGTH(in_str)
    Description
    @@ -118,7 +118,7 @@
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to find the length of.
     @@ -148,7 +148,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - LOWER(input_string) + LOWER(in_str)
    Description
    @@ -160,7 +160,7 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to convert to lower case.
     @@ -188,7 +188,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - LTRIM(input_string [, characters ]) + LTRIM(in_str [, char])
    Description
    @@ -202,14 +202,13 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to remove the leading characters from.
     -
    characters
    -
    Optional argument, if unspecified then this will default to whitespace - (i.e. " ").
    +
    char
    +
    [Optional; default is whitespace, i.e. " "]
    A string, or any valid expression which evaluates to a string, that represents the characters to trim from the input string. Each character in this string will be trimmed from the input string, it is therefore @@ -247,7 +246,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - POSITION(input_string, search_string) + POSITION(in_str, search_str)
    Description
    @@ -260,12 +259,12 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to search within.
     -
    search_string
    +
    search_str
    A string, or any valid expression which evaluates to a string, that is the string to search for.
     @@ -298,7 +297,7 @@ LENGTH("N1QL is awesome") as n1ql;

    Returns

    - REPEAT(input_string, n) + REPEAT(in_str, n)
    Description
    @@ -310,12 +309,12 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to repeat.
     -
    n
    +
    n
    An integer, or any valid expression which evaluates to an integer, that is the number of times to repeat the string.
     @@ -352,7 +351,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - REPLACE(input_string, search_string, replacement [, n ]) + REPLACE(in_str, search_str, replace [, n ])
    Description
    @@ -363,24 +362,23 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to search for replacements in.
     -
    search_string
    +
    search_str
    A string, or any valid expression which evaluates to a string, that is the string to replace.
     -
    replacement
    +
    replace
    A string, or any valid expression which evaluates to a string, that is the string to replace the search string with.
    n
    -
    Optional argument, if unspecified then by default all instances of the - search string are replaced.
    +
    [Optional; default is all instances of the search string are replaced]
    An integer, or any valid expression which evaluates to an integer, which represents the number of instances of the search string to replace. If a negative value is specified then all instances of the @@ -414,7 +412,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - REVERSE(input_string) + REVERSE(in_str)
    Description
    @@ -427,7 +425,7 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to reverse.
     @@ -457,7 +455,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - RTRIM(input_string [, characters ]) + RTRIM(in_str [, char])
    Description
    @@ -470,15 +468,14 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to convert to remove trailing characters from.
     -
    characters
    -
    Optional argument, if unspecified then this will default to whitespace - (i.e. " ").
    +
    char
    +
    [Optional; default is whitespace, i.e. " "]
    A string, or any valid expression which evaluates to a string, that represents the characters to trim from the input string. Each character in this string will be trimmed from the input string, it is therefore @@ -517,7 +514,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - SPLIT(input_string [, input_substring ]) + SPLIT(in_str [, in_substr])
    Description
    @@ -529,12 +526,12 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to split.
     -
    input_substring
    +
    in_substr
    A string, or any valid expression which evaluates to a string, that is the substring to split the input string on.
     @@ -577,7 +574,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - SUBSTR(input_string, start_pos [, length ]) + SUBSTR(in_str, start_pos [, length])
    Description
    @@ -590,19 +587,18 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to convert to extract the substring from.
     -
    start_pos
    +
    start_pos
    An integer, or any valid expression which evaluates to an integer, that is the start position of the substring.
     -
    length
    -
    Optional argument, defaults to capturing to the end of the string if - unspecified.
    +
    length
    +
    [Optional; default is to capture to the end of the string]
    An integer, or any valid expression which evaluates to an integer, that is the length of the substring to extract.
     @@ -634,7 +630,7 @@ LENGTH("N1QL is awesome") as n1ql;
    - SUFFIXES(input_string) + SUFFIXES(in_str)
    Description
    @@ -645,7 +641,7 @@ LENGTH("N1QL is awesome") as n1ql;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to generate the suffixes of.
     @@ -716,11 +712,11 @@ AND type="airport";{
    - TITLE(input_string) + TITLE(in_str)

    Alias for INITCAP().

    - TOKENS(input_string, options) + TOKENS(in_str, opt)
    Description
    @@ -736,7 +732,7 @@ AND type="airport";{
    -
    input_string
    +
    in_str
    A valid JSON object, this can be anything: constant literal, simple JSON value, JSON key name or the whole document itself.
    @@ -791,7 +787,7 @@ AND type="airport";{
    -
    options
    +
    opt
    A JSON object indicating the options passed to the TOKENS() function. Options can take the following options, and each invocation of TOKENS() can choose one or more of the options:
    - TRIM(expression [, characters ]) + TRIM(in_str [, char])
    Description
    @@ -935,15 +931,14 @@ AND type = 'hotel' ;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to convert to remove trailing and leading characters from.
     -
    characters
    -
    Optional argument, if unspecified then this will default to whitespace - (i.e. " ").
    +
    char
    +
    [Optional; default is whitespace, i.e. " "]
    A string, or any valid expression which evaluates to a string, that represents the characters to trim from the input string. Each character in this string will be trimmed from the input string, it is therefore @@ -983,7 +978,7 @@ AND type = 'hotel' ;
    - UPPER(input_string) + UPPER(in_str)
    Description
    @@ -994,7 +989,7 @@ AND type = 'hotel' ;
    -
    input_string
    +
    in_str
    A string, or any valid expression which evaluates to a string, that is the string to convert to upper case.
     diff --git a/content/n1ql/n1ql-language-reference/union.dita b/content/n1ql/n1ql-language-reference/union.dita index 7d0e9d4ea0..f4508b3f58 100644 --- a/content/n1ql/n1ql-language-reference/union.dita +++ b/content/n1ql/n1ql-language-reference/union.dita @@ -1,26 +1,151 @@ - - + + UNION, INTERSECT, and EXCEPT + The UNION, INTERSECT, and EXCEPT + operators combine the resultsets of two or more SELECT statements. - - -

    UNION, INTERSECT, and EXCEPT combine results from multiple subselects.

    - -

    UNION returns values from the first and second subselects.

    - -

    INTERSECT returns values that are present in both the first and second subselects.

    - -

    EXCEPT returns values from the first subselect that are absent from the second subselect.

    - -

    UNION, INTERSECT, and EXCEPT return distinct results, such that there are no duplicates.

    - -

    UNION ALL, INTERSECT ALL, and EXCEPT ALL return all applicable values, - including duplicates. These queries are faster, because they do not compute distinct results.

    -

    You can improve the performance of a query by using covering indexes, where the index includes - all the information needed to satisfy the query. For more information, see +

    Syntax

    + SELECT query1 UNION query2 +| +SELECT query1 INTERSECT query2 +| +SELECT query1 EXCEPT query2 + +
    + +
    Arguments
    +
    +
    + +
    UNION
    +
    Returns all values from both the first and second SELECT + statements.
    +     +
    +
    +
    +
    + +
    INTERSECT
    +
    Returns only values present in both the first and second SELECT + statements.
    +     +
    +
    +
    +
    + +
    EXCEPT
    +
    Returns values from the first SELECT statement that are absent + from the second SELECT statement.
    +     +
    +
    +
    +
    + +
    query1, query2
    +
    Strings or expressions that represent valid SELECT + statements.
    +     +
    +
    +     +
    +

    +

    + +
    Return Values
    +
    UNION, INTERSECT, and EXCEPT return + distinct results, such that there are no duplicates.
    +
    UNION ALL, INTERSECT ALL, and EXCEPT + ALL return all applicable values, including duplicates. These queries are + faster, because they do not compute distinct results.
    +     +
    +

    +

    +

    You can improve the performance of a query by using covering indexes, where the index + includes all the information needed to satisfy the query. For more information, see .

    - - -     -     +

    Examples

    +

    For the following examples, consider these queries and + results.Q1: SELECT DISTINCT city FROM `travel-sample` WHERE type = "airport"; (1641 results) + +Q2: SELECT DISTINCT city FROM `travel-sample` WHERE type = "hotel"; (274 results)

    +

    Example 1: UNION of Q1 and + Q2.SELECT DISTINCT city FROM `travel-sample` WHERE type = "airport" +UNION SELECT DISTINCT city FROM `travel-sample` WHERE type = "hotel";This + gives 1871 + results:[ + { + "city": "Calais" + }, + { + "city": "Peronne" + }, + { + "city": "Nangis" + }, + { + "city": "Bagnole-de-l'orne" + }, +...Example + 2: INTERSECT of Q1 and + Q2.SELECT DISTINCT city FROM `travel-sample` WHERE type = "airport" +INTERSECT SELECT DISTINCT city FROM `travel-sample` WHERE type = "hotel";This + gives 44 + results:[ + { + "city": "Cannes" + }, + { + "city": "Nice" + }, + { + "city": "Orange" + }, + { + "city": "Avignon" + }, +...Example + 3: EXCEPT of Q1 and + Q2.SELECT DISTINCT city FROM `travel-sample` WHERE type = "airport" +EXCEPT SELECT DISTINCT city FROM `travel-sample` WHERE type = "hotel";This + gives 1597 + results:[ + { + "city": "Calais" + }, + { + "city": "Peronne" + }, + { + "city": "Nangis" + }, + { + "city": "Bagnole-de-l'orne" + }, +...Example + 4: EXCEPT of Q2 and + Q1.SELECT DISTINCT city FROM `travel-sample` WHERE type = "hotel" +UNION SELECT DISTINCT city FROM `travel-sample` WHERE type = "airport";This + gives 230 + results:[ + { + "city": "Medway" + }, + { + "city": "Gillingham" + }, + { + "city": "Giverny" + }, + { + "city": "Highland" + }, +...

    + + diff --git a/content/n1ql/n1ql-language-reference/where.dita b/content/n1ql/n1ql-language-reference/where.dita index 0df6bfffcd..b31b539316 100644 --- a/content/n1ql/n1ql-language-reference/where.dita +++ b/content/n1ql/n1ql-language-reference/where.dita @@ -1,51 +1,153 @@ - - + + WHERE clause + The WHERE clause filters resultsets based specified + conditions. - -

    where-clause:

    -WHERE condition -

    condition:

    - expression - -

    The WHERE clause serves to filter a stream of variable bindings, retaining those bindings for - which the truth value of the WHERE clause is TRUE and discarding the bindings for which this - truth value is FALSE.

    -

    Example

    - -SELECT * FROM tutorial WHERE fname = 'Ian' -

    Returns:

    - - { - "results": [ -{ - "age": 56, - "children": [ - { - "age": 17, - "fname": "Abama", - "gender": "m" - }, - { - "age": 21, - "fname": "Bebama", - "gender": "m" - } - ], - "email": "ian@gmail.com", - "fname": "Ian", - "hobbies": [ - "golf", - "surfing" - ], - "lname": "Taylor", - "relation": "cousin", - "title": "Mr.", - "type": "contact" - } - ] - } - -     -     + +
    + Purpose +

    When you want to narrow down your resultset by one or more criteria, use the + WHERE clause to filter your resultset.

    +

    For example, list airports in France, detailed in Example 1:

    + SELECT airportname, city, country +FROM `travel-sample` +WHERE type = "airport" +AND country = "France" +LIMIT 4; +
    +
    + Syntax +

    WHERE cond [ bool_type cond2 ]*

    +
    +
    + Arguments +

    +

    + +
    cond
    +
    [Required] String expression that represents a filter to be applied to the + resultset.
    +     +
    +
    + +
    bool_type
    +
    +
    + +
    AND
    +
    String expression of an inclusive filter to be applied to the resultset.
    +     +
    +
    + +
    OR
    +
    String expression of a union filter to be applied to the resultset.
    +     +
    +
    + +
    NOT
    +
    String expression of an exclusive filter to be applied to the resultset.
    +     +
    +
    +     +
    +

    +
    +
    + Examples +

    Example 1: Use WHERE and AND to filter the + resultset.

    +

    To list only airports that are in France, use the WHERE clause for the + "type" field and the AND clause for the "country' field.

    + SELECT airportname, city, country +FROM `travel-sample` +WHERE type = "airport" +AND country = "France" +LIMIT 4; +

    Results:

    + [ + { + "airportname": "Calais Dunkerque", + "city": "Calais", + "country": "France" + }, + { + "airportname": "Peronne St Quentin", + "city": "Peronne", + "country": "France" + }, + { + "airportname": "Les Loges", + "city": "Nangis", + "country": "France" + }, + { + "airportname": "Couterne", + "city": "Bagnole-de-l'orne", + "country": "France" + } +] +

    Example 2: Use WHERE and OR to filter the + resultset.

    +

    List only the landmarks that start with the letter "C" or "K". Note that the first position + of the SUBSTR + function is 0.

    + SELECT name +FROM `travel-sample` +WHERE type = "landmark" +AND ( CONTAINS(SUBSTR(name,0,1),"C") + OR CONTAINS(SUBSTR(name,0,1),"K") ) +LIMIT 4; +

    Results:

    + [ + { + "name": "City Chambers" + }, + { + "name": "Kingston Bridge" + }, + { + "name": "Clyde Arc" + }, + { + "name": "Clyde Auditorium" + } +] +

    Example 3: Use WHERE and NOT to filter the + resultset.

    +

    List landmark restaurants, except Thai restaurants.

    + SELECT name, activity +FROM `travel-sample` +WHERE type = "landmark" +AND activity = "eat" +AND NOT CONTAINS(name,"Thai") +LIMIT 4; +

    Results:

    + [ + { + "activity": "eat", + "name": "Hollywood Bowl" + }, + { + "activity": "eat", + "name": "Spice Court" + }, + { + "activity": "eat", + "name": "Beijing Inn" + }, + { + "activity": "eat", + "name": "Ossie's Fish and Chips" + } +] +
    + + diff --git a/content/n1ql/n1ql-rest-api/index.dita b/content/n1ql/n1ql-rest-api/index.dita index af9507c0ff..9334acc691 100644 --- a/content/n1ql/n1ql-rest-api/index.dita +++ b/content/n1ql/n1ql-rest-api/index.dita @@ -41,56 +41,60 @@
    Named parameters
    If the statement in a request contains named parameters, the request should contain the - parameters described in the following table.

    Here is an example of a - statement containing named - parameters:

    SELECT detail FROM emp WHERE name = $nval AND age > $aval - - - + parameters described in the following table.

    Example 1: A + statement containing named parameters.

    SELECT detail FROM emp WHERE name = $nval AND age > $aval - - - - + + Parameter Name Value - - - - - - - - - statement - - - SELECT detail FROM emp WHERE name = $nval AND age > - $aval - - - - - $nval - - "smith" - - - - $aval - - 45 - -
    - - - -

    There should be a named parameter in the request for each - query parameter in the request’s statement parameter.

    + + + + + statement + + + SELECT detail FROM emp WHERE name = $nval AND + age > $aval + + + + + $nval + + "smith" + + + + $aval + + 45 + + + + There should be a named parameter in the request for each query + parameter in the request’s statement parameter.
    +
    The % symbol is the escape character in URIs, so when using + % as a wildcard in a query, we need to escape that by + replacing it with its corresponding ASCII code + %25.Example 2: A statement containing a wildcard + parameter.
    +
    $ curl -v http://172.23.99.98:8093/query/service -d 'statement=SELECT meta().id FROM `travel-sample` WHERE type = "hotel" and meta().id LIKE $1 &args=["hotel_1002%25"]'Results:
    +
    + [ + { + "id": "hotel_10026" + }, + { + "id": "hotel_10025" + } +] +
    Positional parameters
    @@ -106,20 +110,22 @@ - + - - statement - - - SELECT detail FROM emp WHERE name = $1 AND age > - $2 - + Parameter Name + Value + + statement + + SELECT detail FROM emp WHERE name = $1 AND + age > $2 + + @@ -139,7 +145,7 @@ - + diff --git a/content/n1ql/n1ql_application_continuity.dita b/content/n1ql/n1ql_application_continuity.dita index 3337870da7..61473b6a4c 100644 --- a/content/n1ql/n1ql_application_continuity.dita +++ b/content/n1ql/n1ql_application_continuity.dita @@ -8,7 +8,7 @@ Before doing any upgrade, you must have all the required RBAC roles of every Data, Index, and Query service involved in the upgrade. For details about RBAC permissions, see .This page refers to the new 5.0 API as "API-2" and the pre-5.0 API as "API-1".

    API-1 and API-2 is used only between Index nodes and Query nodes since the Data nodes diff --git a/content/operator/accessingCouchbase.html b/content/operator/accessingCouchbase.html new file mode 100644 index 0000000000..366bacca60 --- /dev/null +++ b/content/operator/accessingCouchbase.html @@ -0,0 +1,502 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + +

    + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Accessing Couchbase

    +
    +
    +

    After deploying Couchbase Server on Kubernetes or OpenShift, you can use +the Kubernetes or OpenShift console or CLI to verify that the pods are +all deployed and running. However, to develop, administer, or configure +Couchbase, you need to access it using one of the following:

    +
    +
    + +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/adminConsoleAccess.html b/content/operator/adminConsoleAccess.html new file mode 100644 index 0000000000..449ff5f56d --- /dev/null +++ b/content/operator/adminConsoleAccess.html @@ -0,0 +1,575 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Accessing the Couchbase Web Console

    +
    +

    To access the Couchbase Web Console, you must enable the +exposeAdminConsole setting within the Couchbase cluster configuration +file, for example:

    +
    +
    +
    +
    kind: "CouchbaseCluster"
+spec:
+  ....
+  exposeAdminConsole: true
    +
    +
    +
    +

    This creates a NodePort service which exposes the Web Console on each of +your nodes. You can get the port that the service is exposing to your +nodes by checking the status of the cluster resource. Use +kubectl describe cbc on Kubernetes or oc describe cbc on OpenShift:

    +
    +
    +
    +
    Status:
+  Admin Console Port:       30239
+  Admin Console Port SSL:   31628
    +
    +
    +
    +

    The Web Console is now available across all the nodes in your cluster +via <node_ip>:30239, where <node_ip> represents the IP address of a +Kubernetes node. The username and password for Couchbase on port 8091 +are set in your secret.yaml file.

    +
    +
    +

    If at any time you want to remove access to the Web Console, set the +exposeAdminConsole setting to false or remove it from the cluster +configuration file.

    +
    +
    +

    Accessing Specific Couchbase Services

    +
    +

    By default, the exposed port will forward to any of the pods in your +cluster. To access the Web Console of a pod running a specific service +such as search or query, specify which services to expose by using the +adminConsoleServices setting. For example, to expose only pods running +the query service:

    +
    +
    +
    +
    kind: "CouchbaseCluster"
+spec:
+  ....
+  exposeAdminConsole: true
+  adminConsoleServices:
+    - query
    +
    +
    +
    +
    +

    Accessing Specific Couchbase Pods

    +
    +

    To directly access the Web Console of a specific pod, run the following +command on the pod you want to expose.

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    kubectl port-forward cb-example-0000 8091:8091
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    oc port-forward cb-example-0000 8091:8091
    +
    +
    +
    +

    The Web Console for pod cb-example-0000 is now available on +127.0.0.1:8091. Note that you will not be able to access other nodes +in your cluster as this command will only proxy you through to the +specified node.

    +
    +
    +

    The user name and password for Couchbase on port 8091 are set in your +secret.yaml file.

    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/bootstrapOpenshift.html b/content/operator/bootstrapOpenshift.html new file mode 100644 index 0000000000..e3a900cacc --- /dev/null +++ b/content/operator/bootstrapOpenshift.html @@ -0,0 +1,597 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Using OpenShift Templates with Couchbase Operator

    +
    +
    +

    DISCLAIMER

    +
    +

    The following automation should only be used in development clusters +since it involves granting privileges to the operator that aren’t ideal +in a production OpenShift cluster.

    +
    +
    +
    +

    About OpenShift Templates

    +
    +

    OpenShift offers an application catalog that provides templated +deployment of applications into projects. By using templates to deploy +applications, common configurations can be parameterized so that you can +customize instances of your application directly from the user +interface.

    +
    +
    +
    +

    Bootstrapping OpenShift for Couchbase

    +
    +

    Since templates can be imported using the standard oc cli tool, we’ve +created a bootstrap script to assist with importing Couchbase templates +into an OpenShift cluster.

    +
    +
    +
    +
    $ wget https://packages.couchbase.com/kubernetes/0.8.1-beta2/openshift-bootstrap.zip
+$ unzip openshift-bootstrap.zip
+$ cd openshift-bootstrap
+$ chmod +x openshift-bootstrap.sh
+$ ./openshift-bootstrap.sh
    +
    +
    +
    +

    Note: The bootstrap will also create a utility project named +admin-ct-ctl. This utility project consists of a controller that +detects when a new Couchbase cluster is being created from your +template, and ensures that there is an instance of the +couchbase-operator within your clusters project.

    +
    +
    +

    Refreshing the application catalog page should reveal a new Couchbase +template in the catalog. Click the Couchbase template to deploy a +cluster into a new or existing project. Upon completion, you will see a +link to the Couchbase Server Web Console of the cluster that is being +created. This link is also available in the clusters project under +Resources > Routes for future referencing.

    +
    +
    +
    +

    Troubleshooting/FAQ

    +
    +

    In general, it’s important to pay attention to the output from the +bootstrap script, as any failures there indicate that something has gone +wrong.

    +
    +
    +

    The template doesn’t appear in the catalog

    +
    +

    Templates are stored in the OpenShift namespace. Try listing the +templates to ensure that it exists:

    +
    +
    +
    +
    oc get template -n openshift | grep couchbase
    +
    +
    +
    +

    If the template is listed there, then you may need to wait for the +catalog to sync. If not, then you may need to debug your +template +service broker for any issues.

    +
    +
    +
    +

    The Couchbase cluster isn’t starting

    +
    +

    Make sure that the couchbase-operator was deployed in your project. You +should see it in the overview page of your project within the OpenShift +UI. If the operator exists, refer to the documentation about +logs and troubleshooting of the operator. +If the operator does not exist, then check the admin-ct-ctl project.

    +
    +
    +

    As previously mentioned, admin-ct-ctl is the project that is +responsible for deploying the operator into your project. Opening this +project should show a deployment named +couchbase-operator-admin-controller. Check the logs of its pod for +debugging any errors that may occur. If you do not see the deployment +there, then it’s possible that you did not have permissions to create +new projects. Double check the output of the bootstrap script to +confirm.

    +
    +
    +
    +

    Routing to the Couchbase Server Web Console isn’t working

    +
    +

    In some cases, it can take a few minutes for the Couchbase pods to +start, which will delay access to the Couchbase Server Web Console. +Therefore, it’s best to check that the cluster pods are indeed up and +running before assuming its availability. If the pods are running, then +it’s possible that the IP being routed to is not actually publicly +accessible, or that it is associated with the wrong interface.

    +
    +
    +

    Note that the IP address of the route is based on the response from +oc config. If it has gathered the wrong IP address you will need to +edit the route to your cluster via Resources > Routes in the OpenShift +UI.

    +
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/cbopctl.html b/content/operator/cbopctl.html new file mode 100644 index 0000000000..e64d27eab8 --- /dev/null +++ b/content/operator/cbopctl.html @@ -0,0 +1,637 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Ensuring Valid CouchbaseCluster Configurations with cbopctl

    +
    +
    +

    cbopctl is a command line tool that is provided with the Couchbase +Operator that implements a custom subset of the kubectl and oc +commands to ensure that CouchbaseCluster objects are valid +configurations before they are uploaded to Kubernetes. It is important +to check that a configuration is valid before uploading it, because once +a configuration is uploaded to Kubernetes, the Couchbase Operator is +unable to handle configuration errors.

    +
    +
    +

    The following diagram shows the process of uploading a CouchbaseCluster +configuration to Kubernetes using just kubectl:

    +
    +
    +

    Figure 1

    +
    +
    +
      +
    1. +

      The kubectl command is run and the CouchbaseCluster configuration +is sent to the Kubernetes API Server.

      +
      2. +
    2. +

      The CouchbaseCluster configuration undergoes partial validation by +Kubernetes.

      +
      3. +
    3. +

      If the CouchbaseCluster configuration is found to be valid by +Kubernetes, it is inserted into etcd. If it is not valid, an error is +returned to kubectl.

      +
      4. +
    4. +

      etcd acknowledges to the API Server that the CouchbaseCluster +object was persisted.

      +
      5. +
    5. +

      A success message is returned to kubectl.

      +
      6. +
    6. +

      The CouchbaseCluster object is sent to the Couchbase Operator.

      +
      7. +
    +
    +
    +

    Since Kubernetes can only do a partial validation, the only place to do +full validation is in the API Server, because errors in the +configuration must be caught before the CouchbaseCluster configuration +makes it to etcd.

    +
    +
    +

    Kubernetes is a constantly evolving project and the ability to validate +Custom Resource Definitions is still being improved. In Kubernetes 1.8, +for example, there is no ability to do validation. In Kubernetes 1.9, +there is some CRD validation, but it is incomplete and doesn’t allow for +the defaulting of configuration parameters. In Kubernetes 1.11, +defaulting will be added, but there still won’t be support for more +complex types of validation.

    +
    +
    +

    Until Kubernetes provides full native validation, the Couchbase Operator +needs to provide a way to validate that configurations are correct +before submitting them to Kubernetes. The cbopctl command contains all +of Couchbase’s validation code in one command line tool to ensure that +invalid configurations can’t make it into Kubernetes.

    +
    +
    +

    Downloading

    +
    +

    OS X:

    +
    +
    +
    +
    $ curl -O https://packages.couchbase.com/kubernetes/0.8.1-beta2/darwin/cbopctl
+$ chmod +x cbopctl
    +
    +
    +
    +

    Linux:

    +
    +
    +
    +
    $ curl -O https://packages.couchbase.com/kubernetes/0.8.1-beta2/linux/cbopctl
+$ chmod +x cbopctl
    +
    +
    +
    +

    Windows:

    +
    +
    +
    +
    $ curl -O https://packages.couchbase.com/kubernetes/0.8.1-beta2/windows/cbopctl
+$ chmod +x cbopctl
    +
    +
    +
    +
    +

    cbopctl commands

    +
    +

    cbopctl implements the create, delete, and apply commands in a similar +way to kubectl and oc.

    +
    +
    +
    +
    $ cbopctl
+cbopctl [<command>] [<args>]
+
+  apply    Update a Couchbase Cluster
+  create   Create a new Couchbase Cluster
+  delete   Delete a new Couchbase Cluster
+
+Optional Flags:
+
+     --version                Prints version information
+  -h,--help                   Prints the help message
    +
    +
    +
    +

    The create command should be used to create new CouchbaseClusters, +delete can be used to delete CouchbaseClusters (although no validation +takes place during a delete so this command is here for completeness), +and apply should be used when updating a cluster.

    +
    +
    +

    Each subcommand has three flags - one that is required and two that are +optional.

    +
    +
    +
    +
    Required Flags:
+
+  -f,--filename               Filename or the resource to create
+
+Optional Flags:
+
+     --dry-run                If true, only print the object that would be sent,
+                              without sending it
+     --kubeconfig             The path to your Kubernetes configuration
+  -h,--help                   Prints the help message
    +
    +
    +
    +

    Specifying the filename of a CouchbaseCluster configuration is required +for all commands. The dry-run flag can be used to run validation on +the specified configuration file without uploading it to Kubernetes. +This is particularly useful if you want to use features in kubectl +that are not in cbopctl, but you still want to validate the +configuration before pushing it to Kubernetes. The kubeconfig flag is +also available in case the configuration to your Kubernetes cluster is +located on a non-default path.

    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/clusterStatusGuide.html b/content/operator/clusterStatusGuide.html new file mode 100644 index 0000000000..92e0d0922e --- /dev/null +++ b/content/operator/clusterStatusGuide.html @@ -0,0 +1,839 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    CouchbaseCluster Status

    +
    +
    +

    Kubernetes users often need to check the status of various objects (such +as pods, deployments, and StatefulSets) that have been deployed inside +their Kubernetes clusters. Status checking is done by describing one +of the objects in the cluster using kubectl describe command.

    +
    +
    +

    The kubectl describe command outputs the configuration for the +specified object, a status section, and an event section. The status +section varies depending on the object type and shows various health +metrics related to the object described. The events sections is printed +last and shows major events that have happened during the life of the +object.

    +
    +
    +

    Since the Couchbase Operator registers a CouchbaseCluster Custom +Resource Definition (CRD) with Kubernetes, it allows Kubernetes to know +about Couchbase clusters natively. This means that the CouchbaseCluster +becomes another object that can be described to get the configuration, +status, and events specific to a particular Couchbase cluster.

    +
    +
    +

    To describe a Couchbase cluster named cb-example, run the following +command:

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    kubectl describe couchbasecluster cb-example
    +
    +
    +
    +

    Note that the object name in the above command is couchbasecluster. +You can also use the shorthand name cbc as the object name to save +some typing.

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    kubectl describe cbc cb-example
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    oc describe cbc cb-example
    +
    +
    +
    +

    Below is an example of the output of the kubectl describe command. The +various parts of the output and their significance will be discussed in +detail in the following sections.

    +
    +
    +
    +
    Name:       cb-example
+Kind:       CouchbaseCluster
+Spec:   ...
+Status:
+  Admin Console Port:       30239
+  Admin Console Port SSL:   31628
+  Buckets:
+    Default:
+      Conflict Resolution:  seqno
+      Enable Flush:     true
+      Enable Index Replica: false
+      Eviction Policy:      fullEviction
+      Io Priority:      high
+      Memory Quota:     128
+      Name:         default
+      Replicas:         1
+      Type:         couchbase
+  Cluster Id:           9ae7fe4634e0360cf9c9245cb4ebb27b
+  Conditions:
+    ...
+    Message:            Data is equally distributed across all nodes in the cluster
+    Reason:         Cluster is balanced
+    Status:         True
+    Type:           Balanced
+  Control Paused:       false
+  Current Version:      enterprise-5.0.1
+  Members:
+    Ready:
+      cb-example-0000
+      cb-example-0001
+      cb-example-0002
+  Phase:        Running
+  Reason:
+  Size:         3
+  Events:
+  FirstSeen LastSeen    Count   From            Type        Reason          Message
+  --------- --------    -----   ----            --------    ------          -------
+  21m       21m     1   couchbase-operator  Normal      NewMemberAdded      New member cb-example-0000 added to cluster
    +
    +
    +
    +

    Status And Events

    +
    +

    Often when the status spec of the schema is updated, an associated event +is generated which details the reason why the change occurred. For +instance, increasing the size of the cluster causes an add member event +to appear in the Events section of the status result. The following +section explains the information provided in the status response, along +with any events associated with value updates.

    +
    +
    +

    Admin Console

    +
    +
    +
    Admin Console Port: 30239
+Admin Console Port SSL: 31628
    +
    +
    +
    +

    These ports are used for exposing the Couchbase cluster’s web console. +This section of the status is only visible when the exposeAdminConsole +setting of the cluster spec is enabled. See +Accessing the Couchbase Web Console for +information on how to expose and access the Couchbase Server Web +Console.

    +
    +
    +

    Enabling and disabling the exposeAdminConsole setting produces the +following events, respectively:

    +
    +
    +
    +
    Events:
+  ...  ServiceCreated       Service for admin console `cb-example-ui` was created
+  ...  ServiceDeleted       Service for admin console `cb-example-ui` was deleted
    +
    +
    +
    +
    +

    Buckets

    +
    +
    +
    Buckets:
+  Default:
+    Memory Quota:       128
+    Name:           default
+    ...
    +
    +
    +
    +

    A list of buckets currently active within the Couchbase cluster. The +values of the bucket status are the same as the values provided in the +bucket specification. See +CouchbaseCluster Configuration for +details about the bucket specification. When buckets from the +specification are added, updated, or removed the following events are +generated, respectively:

    +
    +
    +
    +
    Events:
+  ... BucketCreated     A new bucket `default` was created
+  ... BucketEdited      Bucket `default` was edited
+  ... BucketDeleted     Bucket `default` was deleted
    +
    +
    +
    +

    The bucket section of the status only reflects valid changes that have +been made to the cluster. In the case of attempts to update the bucket +section with an invalid specification, the following message will be +seen in the Conditions section of the Status:

    +
    +
    +
    +
    Conditions:
+  Message:  Bucket: default cannot change (default) bucket param='conflictResolution' from 'seqno' to 'timestamp'
+  Reason:       Bucket edit failed
+  Status:       False
+  Type:         ManageBuckets
    +
    +
    +
    +

    To resolve errors resulting from invalid updates, edit the cluster +specification according to the message shown under Conditions and apply +the updated specification.

    +
    +
    +
    +

    Cluster ID

    +
    +
    +
    Cluster Id:         9ae7fe4634e0360cf9c9245cb4ebb27b
    +
    +
    +
    +

    The cluster identifier as provided by the Couchbase cluster. This value +is provided directly by the Couchbase cluster and is therefore never +updated by any changes to the cluster specification.

    +
    +
    +
    +

    Members

    +
    +
    +
    Members:
+  Ready:
+    cb-example-0000
+    cb-example-0001
+    cb-example-0002
    +
    +
    +
    +

    Members represent pods that are managed by the Couchbase Operator. All +members in the Ready section represent pods that make up the Couchbase +cluster. The status of cluster members is directly affected by the value +of Size in the cluster spec. When Size is increased, a member add +event is generated, followed by rebalance:

    +
    +
    +
    +
    Events:
+  ... NewMemberAdded        New member cb-example-0003 added to cluster
+  ... RebalanceStarted  A rebalance has been started to balance data across the cluster
    +
    +
    +
    +

    Removing a member generates the expected removal event, followed by a +rebalance:

    +
    +
    +
    +
    Events:
+  ... RebalanceStarted  A rebalance has been started to balance data across the cluster
+  ... MemberRemoved     Existing member cb-example-0003 removed from the cluster
    +
    +
    +
    +

    Note: It is also possible for a MemberRemoved event to be generated +when an auto-failover occurs and a member is replaced by a new member.

    +
    +
    +

    The Conditions section of the status is also updated as the operator +works to resolve changes to cluster spec size or failures. Important +conditions to check when scaling are:

    +
    +
    +
      +
    • +

      Balanced: Denoting whether data is equally distributed across all +nodes in the cluster.

      +
      • +
    • +

      Available: Denoting whether all members are up and all vBuckets are +available.

      +
      • +
    • +

      Scaling: Denoting whether the cluster is currently scaling.

      +
      • +
    +
    +
    +

    See Conditions and Events for more +information about these conditions and their statuses.

    +
    +
    +
    +

    Phase

    +
    +
    +
    Phase:      Running
    +
    +
    +
    +

    The current phase of the cluster displayed as one of the following +phases:

    +
    +
    +
      +
    • +

      Creating: When a cluster is first deployed.

      +
      • +
    • +

      Running: After the couchbasecluster resource object is available and +the first orchestrator pod is running.

      +
      • +
    • +

      Failed: When a failure occurs within the creation or running phase of +the cluster.

      +
      • +
    • +

      None: Initial state denoted by an empty string.

      +
      • +
    +
    +
    +
    +

    Size

    +
    +
    +
    Size:           3
    +
    +
    +
    +

    The size of the Couchbase cluster. When you change this value in the +cluster specification, this value is also updated as members are added +to the Couchbase cluster. If an error occurs while the Operator is +attempting to conform the cluster to the desired size, check the +Conditions section for information about the current state of the +cluster. See Accessing the Couchbase Web +Console for information on how to manually access the cluster and +collect logs when additional troubleshooting is needed.

    +
    +
    +
    +

    Conditions

    +
    +
    +
    Conditions:
+  ...
+  Message:          Data is equally distributed across all nodes in the cluster
+  Reason:           Cluster is balanced
+  Status:           True
+  Type:         Balanced
    +
    +
    +
    +

    A list of conditions reflecting the current state of the Couchbase +cluster. Each Condition item is denoted by a Type along with an +associated Status. The various conditions and their statuses are +documented in Conditions and Events.

    +
    +
    +
    +

    Events

    +
    +
    +
    Events:
+  29m       29m     1   couchbase-operator-1917615544-j1mg8         Normal      NewMemberAdded      New member cb-example-0003 added to cluster
+  29m       29m     1   couchbase-operator-1917615544-j1mg8         Normal      RebalanceStarted    A rebalance has been started to balance data across the cluster
+  51m       51m     1   couchbase-operator-1917615544-j1mg8         Normal      BucketEdited        Bucket `default` was edited
    +
    +
    +
    +

    Events generated during cluster reconciliation. The last 10 events are +recorded and timestamped as the Operator works to reconcile the cluster +to its desired cluster state. The types of events that can occur +throughout the lifecycle of a CouchbaseCluster are documented in +Conditions and Events.

    +
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/conditionsAndEvents.html b/content/operator/conditionsAndEvents.html new file mode 100644 index 0000000000..020a4cb04f --- /dev/null +++ b/content/operator/conditionsAndEvents.html @@ -0,0 +1,624 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Status Conditions and Events

    +
    +
    +

    The actions of the Couchbase Operator and the state of the cluster are +communicated using the standard Kubernetes convention.

    +
    +
    +

    In Kubernetes, you can normally view more information about an object +using the kubectl describe command which displays, among other things, +the Events +and +Conditions +associated with the resource. Similarly, the Couchbase Operator exposes +the events and conditions for each CouchbaseCluster custom resource. If +you’re using OpenShift, you should be able to do the same using the +oc describe command.

    +
    +
    +

    Events

    +
    +

    The lifecycle of a CouchbaseCluster includes the following events (in no +particular order):

    +
    +
    +
      +
    • +

      A new Couchbase node was added to the cluster.

      +
      • +
    • +

      A Couchbase node was removed from the cluster.

      +
      • +
    • +

      A rebalance has started.

      +
      • +
    • +

      Adding a node to the cluster failed.

      +
      • +
    • +

      A bucket was created.

      +
      • +
    • +

      A bucket was deleted.

      +
      • +
    • +

      A bucket was modified.

      +
      • +
    • +

      The Web Console service was created.

      +
      • +
    • +

      The Web Console service was deleted.

      +
      • +
    +
    +
    +
    +

    Conditions

    +
    +

    The CouchbaseCluster conditions and their statuses are defined in the +following list:

    +
    +
    +
      +
    • +

      Available

      +
      +
        +
      • +

        True: All members are up and all vBuckets are available.

        +
        • +
      • +

        False: One or more members are down and some vBuckets are +unavailable.

        +
        • +
      +
      +
      • +
    • +

      Balanced

      +
      +
        +
      • +

        True: The vBuckets are evenly distributed across the cluster.

        +
        • +
      • +

        False: The vBuckets are not evenly distributed across the cluster.

        +
        • +
      • +

        Unknown: The status is currently unknown.

        +
        • +
      +
      +
      • +
    • +

      Scaling

      +
      +
        +
      • +

        True: Scaling from current members size X to spec.size Y.

        +
        • +
      • +

        False: Reason for failure (e.g., No more nodes to place member due to +anti-affinity.)

        +
        • +
      • +

        Not present

        +
        • +
      +
      +
      • +
    • +

      BucketManage

      +
      +
        +
      • +

        False: Creating, editing, or deleting a bucket failed.

        +
        • +
      • +

        Not present

        +
        • +
      +
      +
      • +
    • +

      ConfigManage

      +
      +
        +
      • +

        False: A configuration change failed.

        +
        • +
      • +

        Not present

        +
        • +
      +
      +
      • +
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/couchbaseCliGuide.html b/content/operator/couchbaseCliGuide.html new file mode 100644 index 0000000000..1a84bdd062 --- /dev/null +++ b/content/operator/couchbaseCliGuide.html @@ -0,0 +1,579 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Accessing the Couchbase CLI

    +
    +
    +

    The Couchbase command-line tool couchbase-cli can be used to interact +with the cluster for performing one off administration tasks such as +creating RBAC users and initiating cbcollect info tasks. If your +clusters Web Console is exposed, you can use the CLI tool from any +machine that can access the IP address of your Kubernetes cluster. +Alternatively, couchbase-cli can be run as a Kubernetes job within the +Kubernetes cluster.

    +
    +
    +

    couchbase-cli to Web Console

    +
    +
    +

    Prerequisites:

    +
    +
    + +
    +
    +

    Use the describe command to retrieve the Web Console port:

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    $ kubectl describe cbc cb-example |  grep "Admin Console Port:"
+   Admin Console Port:      32486
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ oc describe cbc cb-example |  grep "Admin Console Port:"
+   Admin Console Port:      32486
    +
    +
    +
    +

    You can now use couchbase-cli to administer the cluster. For example, +adding an RBAC user:

    +
    +
    +
    +
    $ ./couchbase-cli user-manage -c 192.168.99.100:32486 -u Administrator -p password --rbac-username default --rbac-password password --roles admin --auth-domain local --set
+SUCCESS: RBAC user set
    +
    +
    +
    +

    Note that the cluster is being managed by the Couchbase Operator. +Therefore, certain administration tasks such as adding or removing nodes +and buckets can be reverted by the Operator.

    +
    +
    +

    couchbase-cli as Kubernetes Job

    +
    +
    +

    The most secure way to use the couchbase-cli with your Kubernetes +cluster is to run as batch jobs. This option allows you to hide the user +name and password of your cluster by allowing the use of Secrets. For +example, you can run a collect info job using the same secret provided +in the cluster spec:

    +
    +
    +
    +
    $ kubectl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cli-collect-logs.yaml
+job "collect-info" created
+
+# check status
+$ kubectl get job collect-info
+NAME           DESIRED   SUCCESSFUL   AGE
+collect-info   1         1            10s
    +
    +
    +
    +

    On OpenShift, use the following command:

    +
    +
    +
    +
    $ oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cli-collect-logs.yaml
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/couchbaseClients.html b/content/operator/couchbaseClients.html new file mode 100644 index 0000000000..94d946f7c0 --- /dev/null +++ b/content/operator/couchbaseClients.html @@ -0,0 +1,587 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Using Couchbase Clients

    +
    +
    +

    The Kubernetes networking model poses some challenges when interacting +with a Couchbase cluster. This document outlines some of the ways that +Couchbase clients can connect with the cluster.

    +
    +
    +

    From a Pod in the Same Kubernetes Cluster

    +
    +

    The recommended network model also happens to be the simplest: Run the +client from a pod in the same Kubernetes cluster.

    +
    +
    +

    When a new Couchbase cluster is created, the operator also creates a +headless service which creates A records for each Couchbase node, and +an SRV record for the cluster as a whole.

    +
    +
    +

    See +DNS +for Services and Pods.

    +
    +
    +

    To create a connection to the cluster from within the overlay (as a pod +in the same Kubernetes cluster) you should use the SRV record to perform +service discovery:

    +
    +
    +
    +
    from couchbase.cluster import Cluster
+
+c = Cluster('couchbase://my-cluster.default.svc.cluster.local')
    +
    +
    +
    +

    Where the domain is composed of my-cluster, which is the name of the +CouchbaseCluster resource and the super domain.

    +
    +
    +
    +

    From an External Address

    +
    +

    Your ability to access the cluster from outside of the local Kubernetes +environment is dependent on your network architecture or cloud provider. +This is particularly important if you want to access Couchbase Server +across the public internet, either from another cloud or from +on-premises hardware.

    +
    +
    +

    It is unlikely that DNS-based service discovery will be available due to +the Kubernetes implementation, so you will need to gather a list of pod +IP addresses for the clients to use. If you are able to direct DNS +queries to the remote Kubernetes DNS service you can get a list of A +records by looking up my-cluster.default.svc.cluster.local.

    +
    +
    +

    Although not supported officially, the following sections aim to detail +how to access the cluster in some example scenarios.

    +
    +
    +

    Routed Network

    +
    +

    A routed network involves inter-pod communication to occur via a router +as north-south traffic; for example, 10.0.0.0/24 and 10.0.1.0/24 are +allocated as pod address prefixes on two Kubernetes nodes. Traffic +between them is sent to the default router which then has a routing +table entry for each prefix and forwards matching packets to the correct +Kubernetes node.

    +
    +
    +

    As a result, any traffic passing through the router can be forwarded +directly to the target pod via its IP address. Clients could reside on +the same subnet as the cluster, a separate subnet connected to the +router, or peered with another network entirely via a VPN tunnel for +example.

    +
    +
    +
    +

    Overlay Network

    +
    +

    An overlay network captures packets destined for another Kubernetes +node, encapsulates it (e.g. GRE, VXLAN) then forwards it directly to the +destination node via east-west traffic, where the packet is +decapsulated. This relies on service discovery in order for nodes to +know about their peer addresses and their pod network prefixes.

    +
    +
    +

    The cluster cannot be accessed via node ports because clients do not +support the concept of each node having different ports per service. As +a result any traffic from clients must be tunneled into the Kubernetes +cluster, either to the node or to a pod running in the overlay, then +NATed to avoid asymmetric routing.

    +
    +
    +

    A possible solution could involve a publicly addressable VPN gateway +server which decapsulates remote traffic, forwards to a Kubernetes node +via an IP-in-IP tunnel, which then performs source NAT after routing +into the overlay.

    +
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/couchbaseClusterConfig.html b/content/operator/couchbaseClusterConfig.html new file mode 100644 index 0000000000..4029b3af75 --- /dev/null +++ b/content/operator/couchbaseClusterConfig.html @@ -0,0 +1,1450 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    CouchbaseCluster Configuration

    +
    +
    +

    The following sample CouchbaseCluster configuration file lists all of +the configuration parameters:

    +
    +
    +
    +
    apiVersion: couchbase.database.couchbase.com/v1beta1
+kind: CouchbaseCluster
+metadata:
+  name: cb-example
+  namespace: default
+spec:
+  baseImage: couchbase/server
+  version: enterprise-5.0.1
+  paused: false
+  antiAffinity: true
+  authSecret: my-secret
+  exposeAdminConsole: true
+  adminConsoleServices:
+    - data
+  cluster:
+    dataServiceMemoryQuota: 256
+    indexServiceMemoryQuota: 256
+    searchServiceMemoryQuota: 256
+    indexStorageSetting: default
+    autoFailoverTimeout: 30
+  buckets:
+    - name: default
+      type: couchbase
+      memoryQuota: 1024
+      replicas: 1
+      ioPriority: high
+      eviction-policy: value-eviction
+      conflictResolution: seqno
+      enableFlush: true
+      enableIndexReplica: false
+  servers:
+    - size: 1
+      name: all_services
+      services:
+        - data
+        - index
+        - query
+        - search
+      dataPath: /opt/couchbase/var/lib/couchbase/data
+      indexPath: /opt/couchbase/var/lib/couchbase/data
+      pod:
+        couchabseEnv:
+          ENV1: value
+        resources:
+          limits:
+            cpu: 4GHz
+            memory: 8GiB
+            storage: 100GiB
+          requests:
+            cpu: 2GHz
+            memory: 8GiB
+            storage: 50GiB
+        labels:
+          couchbase_services: all
+        nodeSelector:
+          instanceType: large
+        tolerations:
+          - key: app
+            operator: Equal
+            value: cbapp
+            effect: NoSchedule
+        automountServiceAccountToken: false
    +
    +
    +
    + +
    +
    +

    Top-Level Definitions

    +
    +

    The following are the top-level parameters for a CouchbaseCluster +configuration:

    +
    +
    +
    +
    apiVersion: couchbase.database.couchbase.com/v1beta1
+kind: CouchbaseCluster
+metadata:
+  name: cb-example
+  namespace: default
+spec:
+...
    +
    +
    +
    +

    apiversion

    +
    +

    This field specifies the API Version for our integration. As our +integration changes over time, you can change the API Version whenever +new features are added. For any given release, the API Versions that are +supported by that Operator will be specified in our documentation. We +also always recommend that you upgrade to the latest API Version when +possible.

    +
    +
    +

    Field rules: This field is required and must be set to a valid API +Version. The value of this field cannot be changed after the cluster is +created.

    +
    +
    +
    +

    kind

    +
    +

    This field specifies that this Kubernetes configuration will use the +custom Couchbase controller to manage the cluster.

    +
    +
    +

    Field rules: This field is required and must always be set to +CouchbaseCluster. The value of this field cannot be changed after the +cluster is created.

    +
    +
    +
    +

    metadata

    +
    +

    Allows setting a name for the Couchbase cluster and a namespace that the +cluster should be deployed in.

    +
    +
    +

    Field rules: A name is highly recommended, but not required. If values +are not set in the metadata field then defaults will be set. The value +of name will be autogenerated and the value of namespace will be set +to default. The value of these fields cannot be changed after the +cluster is created.

    +
    +
    +
    +

    spec

    +
    +

    This field contains the specification on how the Couchbase cluster must +be deployed.

    +
    +
    +
    +
    +

    Spec: The Top-Level

    +
    +

    This section describes the top-level parameters related to a Couchbase +cluster deployment.

    +
    +
    +
    +
    ...
+  baseImage: couchbase/server
+  version: enterprise-5.0.1
+  paused: false
+  antiAffinity: true
+  authSecret: my-auth-secret
+  exposeAdminConsole: true
+  adminConsoleServices:
+    - data
+...
    +
    +
    +
    +

    baseImage

    +
    +

    This field specifies the image that should be used.

    +
    +
    +

    Field rules: The baseImage field is required and should be set to +couchbase/server unless you want Kubernetes to pull the Couchbase +Server docker container from a different location. The value of this +field cannot be changed after the cluster is created.

    +
    +
    +
    +

    version

    +
    +

    This field specifies the version number of Couchbase Server to install. +This should match an available version number in the Couchbase Docker +repository. It may never be changed to a lower version number than the +version that is currently running.

    +
    +
    +

    Field rules: The version field is required and should be set to the +version of Couchbase Server that should be used to build the cluster. At +the time of this writing, the Couchbase Operator has only been tested +with enterprise-5.0.1. The value of this field cannot be changed after +the cluster is created.

    +
    +
    +
    +

    paused

    +
    +

    This field specifies whether or not the Operator is currently managing +this cluster. This parameter should generally be set to true, but may +be set to false if you decide to make manual changes to the cluster. +By disabling the Operator you can change the cluster configuration +without having to worry about the Operator reverting the changes. +However, before re-enabling the Operator, ensure that the Kubernetes +configuration matches the cluster configuration.

    +
    +
    +

    Field rules: The paused field is not required and defaults to +false if not specified.

    +
    +
    +
    +

    antiAffinity

    +
    +

    This field specifies whether or not two pods in this cluster can be +deployed on the same Kubernetes node. In a production setting this +parameter should always be set to true in order to reduce the chance +of data loss in case a Kubernetes node crashes.

    +
    +
    +

    Field rules: The antiAffinity field is not required and defaults to +false if not specified. The value of this field cannot be changed +after the cluster is created.

    +
    +
    +
    +

    authSecret

    +
    +

    This field specifies the name of a Kubernetes Secret that should be used +as the user name and password of the Couchbase super-user.

    +
    +
    +

    Field rules: The authSecret field is required and should reference +the name of a +Kubernetes +Secret that already exists. The value of this field cannot be changed +after the cluster is created.

    +
    +
    +
    +

    exposeAdminConsole

    +
    +

    This field specifies whether or not the Couchbase Server Web Console +should be exposed externally. Exposing the web console is done using a +NodePort service and the port for the service can be found in the +describe output when describing this cluster. This parameter may be +changed while the cluster is running and the Operator will +create/destroy the NodePort service as appropriate.

    +
    +
    +

    Field rules: The exposeAdminConsole field is not required and +defaults to false if not specified.

    +
    +
    +
    +

    adminConsoleServices

    +
    +

    When the Couchbase Server Web Console is exposed, the pod that it is +connected to is chosen based on the client IP address. Since the web +console may display slightly different features based on the services +running on the particular pod it is connected to, we let you specify the +services so that the exposed port will only connect to pods that contain +the services specified for this parameter. A list of one or more +services should be provided. If you do not specify any service, then the +web console will be connected to all pods in the cluster.

    +
    +
    +

    Field rules: The adminConsoleServices list is not required and +defaults to an empty list. An empty list means any node in the cluster +may be chosen when connecting to the Couchbase Server Web Console.

    +
    +
    +
    +
    +

    Spec: The Cluster Settings

    +
    +

    This section describes the various Couchbase cluster settings. This +section is not meant to encompass every setting that is configurable on +Couchbase. Cluster settings not mentioned here can be managed manually.

    +
    +
    +
    +
    ...
+  cluster:
+    dataServiceMemoryQuota: 4096
+    indexServiceMemoryQuota: 1024
+    searchServiceMemoryQuota: 2048
+    indexStorageType: plasma
+    autoFailoverTimeout: 30
+...
    +
    +
    +
    +

    dataServiceMemoryQuota

    +
    +

    The amount of memory to assign to the data service if it is present on a +specific Couchbase node. The amount of memory is defined in Megabytes +(MB).

    +
    +
    +

    Field rules: The dataServiceMemoryQuota field is required and must +be set to be greater than or equal to 256. Keep in mind that the sum of +all memory quotas must be no more than 80% of a pod’s available memory.

    +
    +
    +
    +

    indexServiceMemoryQuota

    +
    +

    The amount of memory to assign to the index service if it is present on +a specific Couchbase node. The amount of memory is defined in Megabytes +(MB).

    +
    +
    +

    Field rules: The indexServiceMemoryQuota field is required and must +be set to be greater than or equal to 256. Keep in mind that the sum of +all memory quotas must be no more than 80% of a pod’s available memory.

    +
    +
    +
    +

    searchServiceMemoryQuota

    +
    +

    The amount of memory to assign to the search service if it is present on +a specific Couchbase node. The amount of memory is defined in Megabytes +(MB). This parameter defaults to 256 MB if it is not set.

    +
    +
    +

    Field rules: The searchServiceMemoryQuota field is required and must +be set to be greater than or equal to 256. Keep in mind that the sum of +all memory quotas must be no more than 80% of a pod’s available memory.

    +
    +
    +
    +

    indexStorageType

    +
    +

    Specifies the backend storage type to use for the index service. If the +cluster already contains a Couchbase Server instance running the index +service, then this parameter cannot be changed until all Couchbase +instances running the index service are removed.

    +
    +
    +

    Field rules: The indexStorageType is required and must be set to +either plasma or memory-optimized. The value of this field can only +be changed if there are no index nodes in the cluster.

    +
    +
    +
    +

    autoFailoverTimeout

    +
    +

    Specifies the auto-failover timeout in seconds. The Operator relies on +the CouchbaseCluster to auto-failover nodes before removing them, so +setting this field to an appropriate value is important.

    +
    +
    +

    Field rules: The autoFailoverTimeout is required and must be greater +than or equal to 5.

    +
    +
    +
    +
    +

    Spec: Bucket Settings

    +
    +

    This section describes one or more buckets that must be created in the +cluster.

    +
    +
    +

    Note: If you remove an already specified bucket from the +configuration, the Operator will delete that bucket from the cluster.

    +
    +
    +
    +
    ...
+  buckets:
+    - name: default
+      type: couchbase
+      memoryQuota: 1024
+      replicas: 1
+      ioPriority: high
+      evictionPolicy: value-eviction
+      conflictResolution: seqno
+      enableFlush: true
+      enableIndexReplica: false
+...
    +
    +
    +
    +

    name

    +
    +

    This field specifies the name of the bucket. This parameter is required +when specifying a bucket.

    +
    +
    +

    Field rules: The name is required and must be a string using +characters and numbers. The value of this field cannot be changed after +the bucket is created.

    +
    +
    +
    +

    type

    +
    +

    This field specifies the type of bucket to create. This parameter can be +set to couchbase, ephemeral, or memcached. If the type is not +specified, it defaults to couchbase.

    +
    +
    +

    Field rules: The type is required and must be set to either +couchbase, ephemeral, or memcached. The value of this field cannot +be changed after the bucket is created.

    +
    +
    +
    +

    memoryQuota

    +
    +

    This field specifies the amount of memory to allocate to this bucket in +Megabytes (MB). If the quota is not specified, it defaults to 100.

    +
    +
    +

    Field rules: The memoryQuota is required and must be set to greater +than or equal to 100.

    +
    +
    +
    +

    replicas

    +
    +

    This field specifies the number of replicas that should be created for +this bucket. This value may be set between 0 and 3 inclusive. If the +number is not set, it defaults to 1. Note that this parameter has no +effect for the memcached bucket type.

    +
    +
    +

    Field rules: The replicas field is required for buckets with type +couchbase and ephemeral and must be set between 0 and 3. Memcached +buckets will ignore values in this field. Changing the value of this +field will cause a rebalance to occur.

    +
    +
    +
    +

    ioPriority

    +
    +

    This field sets the IO priority of background threads for this bucket. +This option is valid for Couchbase and Ephemeral buckets only. Memcached +buckets will ignore values in this field.

    +
    +
    +

    Field rules: The ioPriority is required for buckets with type +couchbase and ephemeral and must be set to either high or low. +Memcached buckets will ignore values in this field. Changing the value +of this field will cause downtime while the bucket is restarted.

    +
    +
    +
    +

    evictionPolicy

    +
    +

    This field sets the memory-cache eviction policy for this bucket. This +option is valid for Couchbase and Ephemeral buckets only.

    +
    +
    +

    Couchbase buckets support either valueOnly or fullEviction. +Specifying the valueOnly policy means that each key stored in this +bucket must be kept in memory. This is the default policy; using this +policy improves the performance of key-value operations, but limits the +maximum size of the bucket. Specifying the fullEviction policy means +that the performance is impacted for key-value operations, but the +maximum size of the bucket is unbounded.

    +
    +
    +

    Ephemeral buckets support either noEviction or nruEviction. +Specifying noEviction means that the bucket will not evict items from +the cache if the cache is full. This type of eviction policy should be +used for in-memory database use cases.

    +
    +
    +

    Specifying nruEviction means that items not recently used will be +evicted from memory when all the memory in the bucket is used. This type +of eviction policy should be used for caching use cases.

    +
    +
    +

    Field rules: The evictionPolicy is required for buckets with type +couchbase and ephemeral. Memcached buckets will ignore values in +this field. Changing the value of this field will cause downtime while +the bucket is restarted.

    +
    +
    +
    +

    conflictResolution

    +
    +

    This field specifies the bucket’s conflict resolution mechanism which is +to be used if a conflict occurs during cross-datacenter replication +(XDCR). There are two supported mechanisms: sequence-based and +timestamp-based.

    +
    +
    +

    The sequence-based conflict resolution mechanism selects the document +that has been updated the greatest number of times since the last sync. +For example, if one cluster has updated a document twice since the last +sync, and the other cluster has updated the document three times, the +document updated three times is selected regardless of the specific +times at which these updates took place.

    +
    +
    +

    The timestamp-based conflict resolution mechanism selects the document +with the most recent timestamp. This is only supported when all of the +clocks on all of the nodes are fully synchronized.

    +
    +
    +

    Field rules: The conflictResolution field is required for buckets +with type couchbase and ephemeral and can be set to either seqno +or lww. Memcached buckets will ignore values in this field. The value +of this field cannot be changed after the bucket has been created.

    +
    +
    +
    +

    enableFlush

    +
    +

    This field specifies whether or not to enable the flush command on this +bucket. This parameter defaults to false if it is not specified. This +parameter only affects Couchbase and Ephemeral buckets.

    +
    +
    +

    Field rules: The enableFlush field can be set to either true or +false. If this parameter is not specified, it defaults to false.

    +
    +
    +
    +

    enableIndexReplicas

    +
    +

    This field specifies whether or not to enable view index replicas for +this bucket. This parameter defaults to false if it is not specified. +This parameter only affects Couchbase buckets.

    +
    +
    +

    Field rules: The enableIndexReplicas field is required for buckets +with type couchbase and can be set to either true or false. +Memcached and Ephemeral buckets will ignore values in this field.

    +
    +
    +
    +
    +

    Spec: Servers

    +
    +

    You must specify at least one - but possibly multiple - node +specifications. A node specification is used to allow multi-dimensional +scaling of a Couchbase cluster with Kubernetes.

    +
    +
    +
    +
    ...
+  servers:
+  - size: 1
+    name: all_services
+    services:
+      - data
+      - index
+      - query
+      - search
+    dataPath: /opt/couchbase/var/lib/couchbase/data
+    indexPath: /opt/couchbase/var/lib/couchbase/data
+    pod:
+      couchabseEnv:
+        CB_ENV_VAR: value
+      resources:
+        limits:
+          cpu: 4GHz
+          memory: 8GiB
+          storage: 100GiB
+        requests:
+          cpu: 2GHz
+          memory: 8GiB
+          storage: 50GiB
+      labels:
+        couchbase_services: all
+      nodeSelector:
+        instanceType: large
+      tolerations:
+        - key: app
+          operator: Equal
+          value: cbapp
+          effect: NoSchedule
    +
    +
    +
    +

    Specification rules: The servers portion of the specification is +required and must always contain at least one server definition. There +must also be at least one definition that contains the data service.

    +
    +
    +

    size

    +
    +

    This field specifies the number of nodes of this type that should be in +the cluster. This allows the user to scale up different parts of the +cluster as necessary. If this parameter is changed at runtime, the +Operator will automatically scale the cluster.

    +
    +
    +

    Field rules: The size is required and can be set to greater than or +equal to 1.

    +
    +
    +
    +

    name

    +
    +

    This field specifies a name for this group of servers.

    +
    +
    +

    Field rules: The name field is required and must be unique in +comparison to the name field of other server definitions. The value of +this field cannot be changed after a server has been defined.

    +
    +
    +
    +

    services

    +
    +

    This field specifies a list of services that should be run on nodes of +this type. Users can specify data, index, query, and search in +the list. At least one service must be specified and all clusters must +contain at least one node specification that includes the data service.

    +
    +
    +

    Field rules: The services list is required and must contain at least +one service. Valid values for services are data, index, query, and +search. The values of this list cannot be changed after a server has +been defined.

    +
    +
    +
    +

    dataPath

    +
    +

    This field specifies the path where data from the data service should be +stored. This parameter is required.

    +
    +
    +

    Field rules: The dataPath is required and be set to a valid path. +The value of this field cannot be changed after a server has been +defined.

    +
    +
    +
    +

    indexPath

    +
    +

    This field specifies the path were data created by indexes should be +stored. This parameter is required.

    +
    +
    +

    Field rules: The indexPath is required and be set to a valid path. +The value of this field cannot be changed after a server has been +defined.

    +
    +
    +
    +

    The Pod Policy

    +
    +

    The pod policy defines settings that apply to all pods deployed with +this node configuration. A pod always contains a single running instance +of Couchbase Server.

    +
    +
    +
    +
    ...
+    pod:
+      couchbaseEnv:
+        CB_ENV_VAR: value
+      resources:
+        limits:
+          cpu: 4GHz
+          memory: 8GiB
+          storage: 100GiB
+        requests:
+          cpu: 2GHz
+          memory: 8GiB
+          storage: 50GiB
+      labels:
+        couchbase_services: all
+      nodeSelector:
+        instanceType: large
+      tolerations:
+        - key: app
+          operator: Equal
+          value: cbapp
+          effect: NoSchedule
    +
    +
    +
    +
    couchbaseEnv
    +
    +

    This field specifies the environment variables (as key-values pairs) +that should be set when the pod is started. The key in this case is the +environment variable name and the value is the value of the environment +variable. This section is optional.

    +
    +
    +

    Field rules: The value of this field cannot be changed after a server +has been defined.

    +
    +
    +
    +
    Resources
    +
    +
    +
    ...
+    resources:
+      limits:
+        cpu: 2GHz
+        memory: 8GiB
+        storage: 100GiB
+      requests:
+        cpu: 1.5GHz
+        memory: 8GiB
+        storage: 50GiB
+...
    +
    +
    +
    +

    limits

    +
    +
    +

    This field lets you reserve resources on a specific node. It defines the +maximum amount of CPU, memory, and storage the pods created in this node +specification will reserve.

    +
    +
    +

    requests

    +
    +
    +

    This field lets you reserve resources on a specific node. The requests +section defines the minimum amount of CPU, memory, and storage the pods +created in this node specification will reserve.

    +
    +
    +

    See the Kubernetes documentation for further information about +managing +compute resources.

    +
    +
    +
    +
    labels
    +
    +

    Labels are key-value pairs that are attached to objects in Kubernetes. +They are intended to specify identifying attributes of objects that are +meaningful to the user and do not directly imply semantics to the core +system. Labels can be used to organize and select subsets of objects. +They do not need to be unique across multiple objects. This section is +optional.

    +
    +
    +

    Labels added in this section will apply to all pods added in this +cluster. Note that by default we add the following labels to each pod, +which should not be overridden.

    +
    +
    +
    +
    app:couchbase
+couchbase_cluster:<metadata:name>
    +
    +
    +
    +

    In the sample configuration file referenced in this topic, the label +would be couchbase_cluster:cb-example.

    +
    +
    +

    The label for the first pod is of the format: +couchbase_node: <metadata:name>-<gen node id>.

    +
    +
    +

    In the sample configuration file referenced in this topic, the label for +the first pod would be couchbase_node:cb-example-0000.

    +
    +
    +

    For more information, see the Kubernetes documentation about +labels.

    +
    +
    +

    Field rules: The value of this field cannot be changed after a server +has been defined.

    +
    +
    +
    +
    nodeSelector
    +
    +

    This field specifies a key-value map of the constraints on node +placement for pods. For a pod to be eligible to run on a node, the node +must have each of the indicated key-value pairs as labels (it can have +additional labels as well). If this section is not specified, then +Kubernetes will place the pod on any available node. For more +information, see the Kubernetes documentation about +label +selectors.

    +
    +
    +

    Field rules: The value of this field cannot be changed after a server +has been defined.

    +
    +
    +
    +
    tolerations
    +
    +

    This field specifies conditions upon which a node should not be selected +when deploying a pod. From the sample configuration file referenced in +this topic, you can see that any node with a label app:cbapp should +not be allowed to run the pod defined in this node specification. You +might do this if you have nodes dedicated for running an application +using Couchbase where you don’t want the database and application to be +running on the same node. For more information about tolerations, see +the Kubernetes documentation on +taints +and tolerations. The tolerations section is optional.

    +
    +
    +

    Field rules: The value of this field cannot be changed after a server +has been defined.

    +
    +
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/deleteCouchbaseCluster.html b/content/operator/deleteCouchbaseCluster.html new file mode 100644 index 0000000000..564b787994 --- /dev/null +++ b/content/operator/deleteCouchbaseCluster.html @@ -0,0 +1,525 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Deleting a CouchbaseCluster

    +
    +
    +

    You can delete a cluster either by using the cluster configuration file +that you created the cluster with, or by deleting the cluster directly.

    +
    +
    +

    Deleting a Cluster Using the Cluster Configuration File To delete a +cluster using the cluster configuration file, say my-cluster.yaml, run +the following command: On Kubernetes:

    +
    +
    +
    +
    kubectl delete -f my-cluster.yaml
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
        oc delete -f my-cluster.yaml
    +
    +
    +
    +

    Deleting a Cluster Directly To delete a cluster directly, say +my-cluster, run the following command: On Kubernetes:

    +
    +
    +
    +
    kubectl delete couchbasecluster my-cluster
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
        oc delete couchbasecluster my-cluster
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/demo.html b/content/operator/demo.html new file mode 100644 index 0000000000..845f4d8227 --- /dev/null +++ b/content/operator/demo.html @@ -0,0 +1,787 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Cluster Deployment Tutorial

    +
    +
    +

    This tutorial walks you through the steps to deploy a 3-nodes cluster +and load data into the cluster.

    +
    +
    + +
    +
    +

    Prerequisites

    +
    +

    Before proceeding to perform the steps in this tutorial, ensure that +you’ve prepared your Kubernetes cluster to run Couchbase pods. See +Prerequisites and Setup for details. The +tutorial also requires the following artifacts that are bundled with the +Couchbase Operator package:

    +
    +
    +
      +
    • +

      create-user.yaml spec to create secrets for the Couchbase RBAC user.

      +
      • +
    • +

      pillowfight-data-loader.yaml job to load data into your cluster.

      +
      • +
    +
    +
    +
    +

    1. Create a Couchbase Cluster

    +
    +

    On Kubernetes:

    +
    +
    +
    +
    $ cbopctl apply -f  https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cluster.yaml
+couchbasecluster "cb-example" created
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ cbopctl apply -f  https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cluster.yaml
+couchbasecluster "cb-example" created
    +
    +
    +
    +

    Note: cbopctl is a command line tool similar to kubectl or oc. It +allows users to validate that the CouchbaseCluster configuration being +sent to Kubernetes is a valid configuration. See +Ensuring Valid CouchbaseCluster Configurations With +cbopctl for download instructions.

    +
    +
    +
    +

    2. Create a Couchbase RBAC User

    +
    +

    Note: This tutorial will be loading data into Couchbase and hence +requires a Couchbase RBAC user to be created. RBAC users can be created +by directly using the CLI, or as a job within the Kubernetes cluster. +See Accessing the Couchbase CLI for details.

    +
    +
    +

    Option 1: Create a default user using the CLI

    +
    +

    Use the describe command to get the Web Console port:

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    $ kubectl describe cbc cb-example |  grep "Admin Console Port:"
+Admin Console Port:     32486
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ oc describe cbc cb-example |  grep "Admin Console Port:"
+Admin Console Port:     32486
    +
    +
    +
    +

    Create a Couchbase RBAC user by running the following command:

    +
    +
    +
    +
    $ ./couchbase-cli user-manage -c 192.168.99.100:32486 -u Administrator -p password --rbac-username default --rbac-password password --roles admin --auth-domain local --set
+SUCCESS: RBAC user set
    +
    +
    +
    +

    (You can skip to the next step: Loading data)

    +
    +
    +
    +

    Option 2: Create a default user with Kubernetes job

    +
    +

    Just as the admin user has a secret, the RBAC user also requires its own +secret. Create a secret for the RBAC user by running the following +commands:

    +
    +
    +
    +
    secret for user named 'default'
+
+$ echo -n "default" | base64
+ZGVmYXVsdA==
+$ echo -n "password" | base64
+cGFzc3dvcmQ=
+
+$ echo 'apiVersion: v1
+kind: Secret
+metadata:
+  name: cb-user-auth
+type: Opaque
+data:
+  default_user: ZGVmYXVsdA==
+  default_password: cGFzc3dvcmQ=' >  cb-user-auth-secret.yaml
+
+$ kubectl create -f  cb-user-auth-secret.yaml
+secret "cb-user-auth" created
    +
    +
    +
    +

    Now, use the RBAC user’s secret to securely create a Couchbase RBAC user +using a Kubernetes/OpenShift job:

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    $ kubectl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cli-create-user.yaml
+job "create-user" created
+
+$ kubectl get job
+NAME          DESIRED   SUCCESSFUL   AGE
+create-user   1         1            4s
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cli-create-user.yaml
+job "create-user" created
+
+$ oc get job
+NAME          DESIRED   SUCCESSFUL   AGE
+create-user   1         1            4s
    +
    +
    +
    +

    Note: If you are not using the default namespace, you must download +and update the couchbase-cli-create-user.yaml file to reflect your +namespace. For example, if your namespace is myproject, edit the command +field in the YAML file to replace +cb-example-0000.cb-example.default.svc with +cb-example-0000.cb-example.myproject.svc. The updated field will now +look like the following snippet:

    +
    +
    +
    +
    command: ["/bin/sh", "-c", "/couchbase-cli-secure user-manage
+                     -c cb-example-0000.cb-example.myproject.svc
+                     -u {auth.admin.username}
+                     -p {auth.admin.password}
+                     --rbac-username {auth.users.default_user}
+                     --rbac-password {auth.users.default_password}
+                     --roles admin --auth-domain local --set"]
    +
    +
    +
    +

    The name of the secret and its keys are very important as the sample +create-user spec mounts the secrets into a volume.

    +
    +
    +
    +
    https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cli-create-user.yaml
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: create-user
+spec:
+...
+      volumes:
+        - name: cb-admin
+          secret:
+            secretName: cb-example-auth
+        - name: cb-users
+          secret:
+            secretName: cb-user-auth
    +
    +
    +
    +
    +
    +

    3. Loading data

    +
    +

    Use the sample pillowfight job to load items into your cluster. The +following spec loads 10k items into the Couchbase cluster:

    +
    +
    +
    +
    ---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: pillowfight
+spec:
+  template:
+    metadata:
+      name: pillowfight
+    spec:
+      containers:
+      - name: pillowfight
+        image: sequoiatools/pillowfight
+        command: ["cbc-pillowfight",
+                  "-U", "couchbase://cb-example-0000.cb-example.default.svc/default?select_bucket=true",
+                  "-I", "10000", "-B", "1000", "-c", "10", "-t", "1", "-P", "password"]
+      restartPolicy: Never
    +
    +
    +
    +

    Note: If you are not using the default namespace, you must download +and update the pillowfight-data-loader.yaml file to reflect your +namespace. For example, if your namespace is myproject, edit the command +field in the YAML file to replace +cb-example-0000.cb-example.default.svc with +cb-example-0000.cb-example.myproject.svc. The updated field will now +look like the following snippet:

    +
    +
    +
    +
    command: ["cbc-pillowfight",
+
+                  "-U", "couchbase://cb-example-0000.cb-example.myproject.svc/default?select_bucket=true",
+                  "-I", "10000", "-B", "1000", "-c", "10", "-t", "1", "-P", "password"]
    +
    +
    +
    +

    To deploy the pillowfight data loader, run the following command:

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    $ kubectl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/pillowfight-data-loader.yaml
+job "pillowfight" created
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/pillowfight-data-loader-openshift.yaml
+job "pillowfight" created
    +
    +
    +
    +

    Once it completes, you should have 10k items loaded in your cluster.

    +
    +
    +

    See Accessing the Couchbase Web Console for +information about how to access the Couchbase Server Web Console.

    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/deployingCouchbase.html b/content/operator/deployingCouchbase.html new file mode 100644 index 0000000000..f105945f6e --- /dev/null +++ b/content/operator/deployingCouchbase.html @@ -0,0 +1,662 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Deploying a Couchbase Cluster

    +
    +
    +

    Prerequisites Before proceeding to deploy a Couchbase cluster, ensure +that the prerequisites are met and the +Couchbase Operator is up and running.

    +
    +
    +

    Deploying a Cluster Deploying a Couchbase cluster requires creating a +configuration file that describes what the cluster should look like. +Like all Kubernetes configurations, Couchbase clusters are defined using +either YAML or JSON (YAML is preferred by Kubernetes) and then pushed +into Kubernetes. Below is a sample configuration file for a Couchbase +cluster.

    +
    +
    +
    +
    apiVersion: couchbase.database.couchbase.com/v1beta1
+kind: CouchbaseCluster
+metadata:
+  name: cb-example
+spec:
+  baseImage: couchbase/server
+  version: enterprise-5.0.1
+  authSecret: cb-example-auth
+  exposeAdminConsole: true
+  cluster:
+    dataServiceMemoryQuota: 256
+    indexServiceMemoryQuota: 256
+    searchServiceMemoryQuota: 256
+    indexStorageSetting: memory_optimized
+    autoFailoverTimeout: 30
+  buckets:
+    - name: default
+      type: couchbase
+      memoryQuota: 128
+      replicas: 1
+      ioPriority: high
+      evictionPolicy: fullEviction
+      conflictResolution: seqno
+      enableFlush: true
+      enableIndexReplica: false
+  servers:
+    - size: 3
+      name: all_services
+      services:
+        - data
+        - index
+        - query
+        - search
+      dataPath: /opt/couchbase/var/lib/couchbase/data
+      indexPath: /opt/couchbase/var/lib/couchbase/data
    +
    +
    +
    +

    By taking a quick look at this configuration file you can see that it +defines a cluster by specifying the following:

    +
    +
    +
      +
    • +

      Cluster name: cb-example (metadata.name)

      +
      • +
    • +

      Couchbase version: 5.0.1 (spec.version)

      +
      • +
    • +

      Buckets: 1 bucket named default (spec.buckets)

      +
      • +
    • +

      Size: 3 node cluster with all services on each node (spec.servers)

      +
      • +
    • +

      Auth secret: cb-example-auth

      +
      • +
    +
    +
    +

    One thing that’s important to note is the authSecret field. The +Couchbase Operator uses Kubernetes Secrets to create and manage the +Couchbase super-user credentials. As a result, the authSecret field +must refer to secret that contains both a user name and a password +field. For convenience, we provide a sample secret that can be pushed +into your Kubernetes cluster. The secret sets the user name to +Administrator and the password to password. To load this secret into +your Kubernetes cluster, run the following command: On Kubernetes:

    +
    +
    +
    +
    $ kubectl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/secret.yaml
+secret "cb-example-auth" created
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/secret.yaml
+secret "cb-example-auth" created
    +
    +
    +
    +

    The Couchbase Operator configuration is now ready to be pushed to +Kubernetes. To push the Couchbase Operator configuration, run the +following command: On Kubernetes:

    +
    +
    +
    +
    $ cbopctl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cluster.yaml
+couchbasecluster "cb-example" created
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ cbopctl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cluster.yaml
+couchbasecluster "cb-example" created
    +
    +
    +
    +

    Note: cbopctl is a command line tool similar to kubectl or oc. It +allows users to validate that the CouchbaseCluster configuration being +sent to Kubernetes is a valid configuration. See +Ensuring Valid CouchbaseCluster Configurations With +cbopctl for download instructions.

    +
    +
    +

    The Couchbase Operator automatically begins creating the cluster. +Depending on the configuration, creating a cluster can take some time. +You can track the progress of cluster creation using the +kubectl describe command. For details on how to use this command and +an explanation of the output, see Displaying +Information about a Couchbase Cluster.

    +
    +
    +

    Once the cluster has been provisioned, you’ll see that various pods, a +service, and a Couchbase cluster have been created. Since the +configuration above was for a 3-nodes cluster, you should see three pods +created. The Couchbase Operator also creates an internal headless +service that can be used by applications deployed inside the same +Kubernetes namespace to connect to the Couchbase cluster. Run the +following command to see the newly created pods:

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    $ kubectl get pods
+NAME                                  READY     STATUS    RESTARTS   AGE
+cb-example-0000                       1/1       Running   0          1m
+cb-example-0001                       1/1       Running   0          1m
+cb-example-0002                       1/1       Running   0          1m
+couchbase-operator-1917615544-pd4q6   1/1       Running   0          8m
    +
    +
    +
    +

    On OpenShift, run the following command:

    +
    +
    +
    +
    $ oc get pods
    +
    +
    +
    +

    A CouchbaseCluster object is also created for the cluster and can be +used to get health and status information about the cluster. For details +on how to inspect these objects, see Listing and +Describing Pods, Services, and CouchbaseClusters

    +
    +
    +

    For further details on what each field in the CouchbaseCluster +configuration does, as well as a list of all available fields see +CouchbaseCluster Configuration.

    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/images/cb-operator-working.jpg b/content/operator/images/cb-operator-working.jpg new file mode 100644 index 0000000000..0d7498ab81 Binary files /dev/null and b/content/operator/images/cb-operator-working.jpg differ diff --git a/content/operator/images/cbopctl_figure_1.jpg b/content/operator/images/cbopctl_figure_1.jpg new file mode 100644 index 0000000000..0d7498ab81 Binary files /dev/null and b/content/operator/images/cbopctl_figure_1.jpg differ diff --git a/content/operator/images/deploy-blueprint.png b/content/operator/images/deploy-blueprint.png new file mode 100644 index 0000000000..87ba6d77da Binary files /dev/null and b/content/operator/images/deploy-blueprint.png differ diff --git a/content/operator/index.html b/content/operator/index.html new file mode 100644 index 0000000000..ba16ed6415 --- /dev/null +++ b/content/operator/index.html @@ -0,0 +1,586 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Couchbase Operator User Guide

    +
    +
    + +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/installKubernetes.html b/content/operator/installKubernetes.html new file mode 100644 index 0000000000..06d7574d83 --- /dev/null +++ b/content/operator/installKubernetes.html @@ -0,0 +1,672 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Installing on Kubernetes With RBAC

    +
    +
    +

    In this setup guide we will walk through the recommended procedure for +setting up the Couchbase Operator in a Kubernetes cluster that has RBAC +enabled. This guide assumes that you are installing on a new Kubernetes +cluster, but if you have an existing system and need to do a custom +setup, then you should be able to modify a few of the parameters in the +various commands and configuration files in order to install the +Couchbase Operator.

    +
    +
    +

    Creating a ClusterRole

    +
    +

    The first step for installing the Couchbase Operator is to create a +ClusterRole that allows the Operator to access the resources it needs +to run. Since the Couchbase Operator might run in many different +namespaces, it is best to create a ClusterRole because you can assign +that role to a ServiceAccount in any namespace.

    +
    +
    +

    To create the ClusterRole for the Couchbase Operator, run the following +command:

    +
    +
    +
    +
    $ kubectl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/k8s-rbac/cluster-role.yaml
    +
    +
    +
    +

    Note: This role only needs to be created once.

    +
    +
    +
    +

    Creating a ServiceAccount

    +
    +

    After the ClusterRole is created, you need to create a ServiceAccount in +the namespace where you are installing the Couchbase Operator, and then +assign the ClusterRole to that ServiceAccount using a +ClusterRoleBinding. In this guide we will use the default namespace +to create the ServiceAccount.

    +
    +
    +
    +
    $ kubectl create serviceaccount couchbase-operator --namespace default
+$ kubectl create clusterrolebinding couchbase-operator --clusterrole couchbase-operator --serviceaccount default:couchbase-operator
    +
    +
    +
    +

    Note: If you would prefer to use a RoleBinding instead of a +ClusterRoleBinding, then the CRD needs to be installed separately from +the Couchbase Operator. This can be accomplished by removing the +-create-crd argument from the operator.yaml file and uploading the +CouchbaseCluster CRD separately.

    +
    +
    +
    +

    Starting the Operator

    +
    +

    Now that the ServiceAccount is set up with the appropriate permissions, +you can start the Couchbase Operator by running the following command:

    +
    +
    +
    +
    $ kubectl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/k8s-rbac/operator.yaml
    +
    +
    +
    +

    Running this command downloads the Couchbase Operator Docker image that +is specified in the operator.yaml file and creates a deployment +which manages a single instance of the Couchbase Operator. The Couchbase +Operator uses a deployment so that it can restart if the pod it’s +running in dies.

    +
    +
    +

    After you run the kubectl create command, it generally takes less than +1 minute for Kubernetes to deploy the Operator and for the Operator to +be ready to run. Using the following commands, you can check the status +of the Operator and see the status of your deployment.

    +
    +
    +
    +
    $ kubectl get deployments -l app=couchbase-operator
    +
    +
    +
    +

    If you run this command immediately after the operator is deployed, the +output will look something like this:

    +
    +
    +
    +
    NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+couchbase-operator   1         1         1            0           10s
    +
    +
    +
    +

    In this case, the deployment is called couchbase-operator. The DESIRED +field in the output shows that this deployment will create 1 pod running +the Couchbase Operator. The CURRENT field shows that 1 Couchbase +Operator pod has been created. However, the AVAILABLE field indicates +that that pod is not ready yet since its value is 0 and not 1. This +means that the Operator is still establishing a connection to the +Kubernetes master node to allow it to get updates on CouchbaseCluster +objects. Once the Operator has completed this task it will be able to +start managing Couchbase clusters and the status will be shown as +AVAILABLE.

    +
    +
    +

    You should continue to poll the status of the Operator until your output +looks similar to the following output:

    +
    +
    +
    +
    NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+couchbase-operator   1         1         1            1           47s
    +
    +
    +
    +

    You can also verify that the Couchbase Operator has come up successfully +using the following command:

    +
    +
    +
    +
    kubectl get pods -l app=couchbase-operator
    +
    +
    +
    +

    If the Couchbase Operator is up and running successfully, the command +returns an output where the READY field shows 1/1, such as:

    +
    +
    +
    +
    NAME                                  READY   STATUS   RESTARTS   AGE
+couchbase-operator-1917615544-t5mhp   1/1     Running  0          57s
    +
    +
    +
    +

    You can also check the logs to confirm that the Couchbase Operator is up +and running. Look for the message: +CRD initialized, listening for events…​ module=controller.

    +
    +
    +
    +
    $ kubectl logs couchbase-operator-1917615544-t5mhp
+time="2018-04-25T03:01:56Z" level=info msg="Obtaining resource lock" module=main
+time="2018-04-25T03:01:56Z" level=info msg="Starting event recorder" module=main
+time="2018-04-25T03:01:56Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main
+time="2018-04-25T03:02:13Z" level=info msg="I'm the leader, attempt to start the operator" module=main
+time="2018-04-25T03:02:13Z" level=info msg="Creating the couchbase-operator controller" module=main
+time="2018-04-25T03:02:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"default\", Name:\"couchbase-operator\", UID:\"9b86c750-47e7-11e8-866e-080027b2a68d\", APIVersion:\"v1\", ResourceVersion:\"23482\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' couchbase-operator-75ddfdbdb5-bz7ng became leader" module=event_recorder
+time="2018-04-25T03:02:13Z" level=info msg="CRD initialized, listening for events..." module=controller
+time="2018-04-25T03:02:13Z" level=info msg="starting couchbaseclusters controller"
    +
    +
    +
    +
    +

    Uninstall

    +
    +

    Uninstalling the Couchbase Operator is a two-step process.

    +
    +
    +
      +
    1. +

      Delete the Couchbase Operator

      +
      +
      +
      $ kubectl delete deployment couchbase-operator
      +
      +
      +
      2. +
    2. +

      Delete the CRD

      +
      +

      Make sure all instances of the Couchbase Operator have been deleted from +the Kubernetes cluster before you delete the CRD. Once all instances of +the Couchbase Operator have been deleted, run the following command to +delete the CRD:

      +
      +
      +
      +
      $ kubectl delete crd couchbaseclusters.couchbase.database.couchbase.com
      +
      +
      +
      3. +
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/installKubernetesNoRBAC.html b/content/operator/installKubernetesNoRBAC.html new file mode 100644 index 0000000000..d77ac8e4c6 --- /dev/null +++ b/content/operator/installKubernetesNoRBAC.html @@ -0,0 +1,630 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Installing on Kubernetes Without RBAC

    +
    +

    In this setup guide we will walk through the recommended procedure for +setting up the Couchbase Operator in a Kubernetes cluster that does not +have RBAC enabled. This guide is recommended mainly for development +clusters or MiniKube. Running a Kubernetes cluster without RBAC is not +recommended in production.

    +
    +
    +

    Note: If you are using a MiniKube cluster with Kubernetes 1.10 or +later, RBAC is enabled by default. You can either disable RBAC, or +follow the instructions to install the +operator on Kubernetes with RBAC enabled.

    +
    +
    +

    Starting the Operator

    +
    +

    Run the following command to install the Couchbase Operator:

    +
    +
    +
    +
    $ kubectl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/k8s/operator.yaml
    +
    +
    +
    +

    Running this command downloads the Couchbase Operator Docker image that +is specified in the operator.yaml file and creates a deployment +which manages a single instance of the Couchbase Operator. The Couchbase +Operator uses a deployment so that it can restart if the pod it’s +running in dies.

    +
    +
    +

    After you run the kubectl create command, it generally takes less than +1 minute for Kubernetes to deploy the Operator and for the Operator to +be ready to run. Using the following commands, you can check the status +of the Operator and see the status of your deployment.

    +
    +
    +
    +
    $ kubectl get deployments -l app=couchbase-operator
    +
    +
    +
    +

    If you run this command immediately after the operator is deployed, the +output will look something like this:

    +
    +
    +
    +
    NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+couchbase-operator   1         1         1            0           10s
    +
    +
    +
    +

    In this case, the deployment is called couchbase-operator. The DESIRED +field in the output shows that this deployment will create 1 pod running +the Couchbase Operator. The CURRENT field shows that 1 Couchbase +Operator pod has been created. However, the AVAILABLE field indicates +that that pod is not ready yet since its value is 0 and not 1. This +means that the Operator is still establishing a connection to the +Kubernetes master node to allow it to get updates on CouchbaseCluster +objects. Once the Operator has completed this task it will be able to +start managing Couchbase clusters and the status will be shown as +AVAILABLE.

    +
    +
    +

    You should continue to poll the status of the Operator until your output +looks similar to the following output:

    +
    +
    +
    +
    NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+couchbase-operator   1         1         1            1           47s
    +
    +
    +
    +

    You can also verify that the Couchbase Operator has come up successfully +using the following command:

    +
    +
    +
    +
    kubectl get pods -l app=couchbase-operator
    +
    +
    +
    +

    If the Couchbase Operator is up and running successfully, the command +returns an output where the READY field shows 1/1, such as:

    +
    +
    +
    +
    NAME                                  READY   STATUS   RESTARTS   AGE
+couchbase-operator-1917615544-t5mhp   1/1     Running  0          57s
    +
    +
    +
    +

    You can also check the logs to confirm that the Couchbase Operator is up +and running. Look for the message: +CRD initialized, listening for events…​ module=controller.

    +
    +
    +
    +
    $ kubectl logs couchbase-operator-1917615544-t5mhp
+time="2018-04-25T03:01:56Z" level=info msg="Obtaining resource lock" module=main
+time="2018-04-25T03:01:56Z" level=info msg="Starting event recorder" module=main
+time="2018-04-25T03:01:56Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main
+time="2018-04-25T03:02:13Z" level=info msg="I'm the leader, attempt to start the operator" module=main
+time="2018-04-25T03:02:13Z" level=info msg="Creating the couchbase-operator controller" module=main
+time="2018-04-25T03:02:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"default\", Name:\"couchbase-operator\", UID:\"9b86c750-47e7-11e8-866e-080027b2a68d\", APIVersion:\"v1\", ResourceVersion:\"23482\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' couchbase-operator-75ddfdbdb5-bz7ng became leader" module=event_recorder
+time="2018-04-25T03:02:13Z" level=info msg="CRD initialized, listening for events..." module=controller
+time="2018-04-25T03:02:13Z" level=info msg="starting couchbaseclusters controller"
    +
    +
    +
    +
    +
    +

    Uninstall

    +
    +

    Uninstalling the Couchbase Operator is a two-step process.

    +
    +
    +
      +
    1. +

      Delete the Couchbase Operator

      +
      +
      +
      $ kubectl delete deployment couchbase-operator
      +
      +
      +
      2. +
    2. +

      Delete the CRD

      +
      3. +
    +
    +
    +

    Note: Make sure all instances of the Couchbase Operator have been +deleted from the Kubernetes cluster before you delete the CRD.

    +
    +
    +

    Once all instances of the Couchbase Operator have been deleted, run the +following command to delete the CRD:

    +
    +
    +
    +
    $ kubectl delete crd couchbaseclusters.couchbase.database.couchbase.com
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/installOpenShift.html b/content/operator/installOpenShift.html new file mode 100644 index 0000000000..3f5a4e0722 --- /dev/null +++ b/content/operator/installOpenShift.html @@ -0,0 +1,764 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Installing on OpenShift

    +
    +

    In this setup guide we will walk through the recommended procedure for +setting up the Couchbase Operator in an OpenShift project. This guide +assumes that you are installing on a new OpenShift cluster, that you +have access to a user with cluster-admin privileges (system:admin in +this guide), and that you have a standard user without cluster-admin +privileges (developer in this guide). If you have an existing system +and need to do a custom setup, then you should be able to modify a few +of the parameters in the various commands and configuration files in +order to install the Couchbase Operator.

    +
    +
    +

    Create an OpenShift Project

    +
    +

    Before installing the Couchbase Operator into OpenShift, you should +create a project with the developer user. For this guide we will +create a project called operator-example, and later we will make sure +that the developer user has the ability to manage the Couchbase +Operator and Couchbase clusters in this project. To create a project, +run the following commands:

    +
    +
    +
    +
    $ oc login -u developer
+$ oc new-project operator-example
+$ oc logout
    +
    +
    +
    +
    +

    CRD and ClusterRole Creation

    +
    +

    The next step in setting up the Couchbase Operator is to install the +CouchbaseCluster Custom Resource Definition (CRD). The Couchbase +Operator can do this for you automatically, but in OpenShift it’s better +to do it manually since installing CRDs is a cluster-level operation and +requires cluster-level permissions that should not be given to typical +users.

    +
    +
    +

    To install the CRD, log in as the system:admin user and run the +following command:

    +
    +
    +
    +
    $ oc login -u system:admin
+$ oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/openshift/crd.yaml
    +
    +
    +
    +

    Note: You only need to install the CRD once. After that, the Couchbase +Operator can be created in any OpenShift project.

    +
    +
    +

    Once the CRD is installed, you need to create a role that allows the +Operator to access the resources it needs to run. Since the Couchbase +Operator might run in many different projects, it is best to create a +ClusterRole because you can assign that role to a ServiceAccount in +any OpenShift project.

    +
    +
    +

    To create the ClusterRole for the Couchbase Operator, run the following +command as the system:admin user:

    +
    +
    +
    +
    $ oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/openshift/cluster-role-sa.yaml
    +
    +
    +
    +

    Note: You only need to create the ClusterRole once.

    +
    +
    +

    Next, since the developer user needs to be able to create and delete +the Couchbase cluster, you need to create another ClusterRole that gives +permissions to handle management of the CouchbaseCluster resource.

    +
    +
    +

    Run the following command to create the ClusterRole for developer:

    +
    +
    +
    +
    oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/openshift/cluster-role-user.yaml
    +
    +
    +
    +
    +

    Setting Up RBAC for an OpenShift Project

    +
    +

    After the ClusterRole is created, you need to create a ServiceAccount +in the project where you are installing the Couchbase Operator, and then +assign the ClusterRole to that ServiceAccount using a RoleBinding. (A +ServiceAccount is used as opposed to a User for the Couchbase Operator +because ServiceAccounts are meant to be used for processes running in +OpenShift.)

    +
    +
    +
    +
    $ oc create serviceaccount couchbase-operator --namespace operator-example
+$ oc create rolebinding couchbase-operator --clusterrole couchbase-operator --serviceaccount operator-example:couchbase-operator
    +
    +
    +
    +

    Couchbase Server containers must be run by the root user, so you need to +set a policy on the ServiceAccount to allow Couchbase Server pods to be +started by the Couchbase Operator. Use the following command to set the +policy for the couchbase-operator ServiceAccount:

    +
    +
    +
    +
    $ oc adm policy add-scc-to-user anyuid system:serviceaccount:operator-example:couchbase-operator
    +
    +
    +
    +

    Now that the ServiceAccount that the Couchbase Operator uses to run has +been set up, you need to allow the developer user to be able to manage +the CouchbaseCluster resource in the operator-example project.

    +
    +
    +
    +
    $ oc create rolebinding couchbasecluster --clusterrole couchbasecluster --user developer --namespace operator-example
    +
    +
    +
    +

    If you want the developer user to be able to manage CouchbaseCluster +objects in all projects, then you can run the following command instead:

    +
    +
    +
    +
    oc create clusterrolebinding couchbasecluster --clusterrole couchbasecluster --user developer
    +
    +
    +
    +

    Note: To set up a ServiceAccount for the operator in another OpenShift +project, repeat the steps above.

    +
    +
    +
    +

    Starting the Operator

    +
    +

    At this point you can now create the Couchbase Operator with the +developer user in the operator-example project. To start the +Couchbase Operator, run the following command:

    +
    +
    +
    +
    $ oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/openshift/operator.yaml
    +
    +
    +
    +

    Running this command downloads the Couchbase Operator Docker image that +is specified in the operator.yaml file and creates a deployment +which manages a single instance of the Couchbase Operator. The Couchbase +Operator uses a deployment so that it can restart if the pod it’s +running in dies.

    +
    +
    +

    After you run the oc create command, it generally takes less than 1 +minute for OpenShift to deploy the Operator and for the Operator to be +ready to run. Using the following commands, you can check the status of +the Operator and see the status of your deployment.

    +
    +
    +
    +
    $ oc get deployments -l app=couchbase-operator
    +
    +
    +
    +

    If you run this command immediately after the operator is deployed, the +output will look something like this:

    +
    +
    +
    +
    NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+couchbase-operator   1         1         1            0           10s
    +
    +
    +
    +

    In this case, the deployment is called ``couchbase-operator''. The +DESIRED field in the output shows that this deployment will create 1 pod +running the Couchbase Operator. The CURRENT field shows that 1 Couchbase +Operator pod has been created. However, the AVAILABLE field indicates +that that pod is not ready yet since its value is 0 and not 1. This +means that the Operator is still establishing a connection to the +Kubernetes master node to allow it to get updates on CouchbaseCluster +objects. Once the Operator has completed this task it will be able to +start managing Couchbase clusters and the status will be shown as +AVAILABLE.

    +
    +
    +

    You should continue to poll the status of the Operator until your output +looks similar to the following output:

    +
    +
    +
    +
    NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+couchbase-operator   1         1         1            1           47s
    +
    +
    +
    +

    You can also verify that the Couchbase Operator has come up successfully +using the following command:

    +
    +
    +
    +
    oc get pods -l app=couchbase-operator
    +
    +
    +
    +

    If the Couchbase Operator is up and running successfully, the command +returns an output where the READY field shows 1/1, such as:

    +
    +
    +
    +
    NAME                                  READY   STATUS   RESTARTS   AGE
+couchbase-operator-1917615544-t5mhp   1/1     Running  0          57s
    +
    +
    +
    +

    You can also check the logs to confirm that the Couchbase Operator is up +and running. Look for the message: ``starting couchbaseclusters +controller''.

    +
    +
    +
    +
    $ oc logs couchbase-operator-1917615544-t5mhp
+time="2018-04-25T03:01:56Z" level=info msg="Obtaining resource lock" module=main
+time="2018-04-25T03:01:56Z" level=info msg="Starting event recorder" module=main
+time="2018-04-25T03:01:56Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main
+time="2018-04-25T03:02:13Z" level=info msg="I'm the leader, attempt to start the operator" module=main
+time="2018-04-25T03:02:13Z" level=info msg="Creating the couchbase-operator controller" module=main
+time="2018-04-25T03:02:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"default\", Name:\"couchbase-operator\", UID:\"9b86c750-47e7-11e8-866e-080027b2a68d\", APIVersion:\"v1\", ResourceVersion:\"23482\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' couchbase-operator-75ddfdbdb5-bz7ng became leader" module=event_recorder
+time="2018-04-25T03:02:13Z" level=info msg="starting couchbaseclusters controller"
    +
    +
    +
    +
    +
    +

    Uninstall

    +
    +

    Uninstalling the Couchbase Operator is a two-step process.

    +
    +
    +
      +
    1. +

      Delete the Couchbase Operator

      +
      +

      You can delete the Couchbase Operator as the developer or +system:admin user.

      +
      +
      +
      +
      $ kubectl delete deployment couchbase-operator
      +
      +
      +
      +

      Note: Deleting the Couchbase Operator from a namespace will not remove +or affect any Couchbase pods in your Kubernetes cluster.

      +
      +
      2. +
    2. +

      Delete the CRD

      +
      +

      You can only delete the CRD as the system:admin user.

      +
      +
      +

      Make sure the Couchbase Operator has been deleted from all other +OpenShift projects in the cluster before you delete the CRD. Once the +Couchbase Operator has been deleted, run the following command to delete +the CRD:

      +
      +
      +
      +
      $ kubectl delete crd couchbaseclusters.couchbase.database.couchbase.com
      +
      +
      +
      3. +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/listAndDescribe.html b/content/operator/listAndDescribe.html new file mode 100644 index 0000000000..a25ac7d150 --- /dev/null +++ b/content/operator/listAndDescribe.html @@ -0,0 +1,830 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Listing And Describing Resources

    +
    +
    +

    When a Couchbase cluster is deployed, additional Kubernetes resources +such as pods and services are created by the Operator to facilitate its +deployment. All resources originating from the Couchbase Operator are +labeled in order to make it easy to list and describe resources +belonging to a specific cluster. The following sections list and explain +the resources created by the Operator when the cluster is running.

    +
    +
    +

    Note that each of the commands below use kubectl with the default +namespace kubectl -n default …​ when using Kubernetes. If you’ve +deployed your cluster using a different namespace, you will need to +explicitly provide the namespace where your cluster is deployed. +Similarly, if you are using OpenShift, the commands oc use the default +namespace oc -n default …​.

    +
    +
    +

    Operator Deployment

    +
    +

    The Operator is started as a deployment resource. The name of the object +will match the name provided in your deployment specification. (See +Prerequisites and Setup for details.) The +Operator deployment can be listed and described as follows: On +Kubernetes:

    +
    +
    +
    +
    $ kubectl get deployments couchbase-operator
+NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+couchbase-operator   1         1         1            1           27m
+
+$ kubectl describe deployments couchbase-operator
+Name:           couchbase-operator
+Namespace:      default
+Labels:         name=couchbase-operator
+Replicas:       1 desired | 1 updated | 1 total | 1 available | 0 unavailable
+Pod Template:
+  Labels:   name=couchbase-operator
+  Containers:
+   couchbase-operator:
+    Image:  couchbase/couchbase-operator:v1
+    Port:   8080/TCP
+    ...
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ oc get deployments couchbase-operator
+NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
+couchbase-operator   1         1         1            1           27m
+
+$ oc describe deployments couchbase-operator
+Name:           couchbase-operator
+Namespace:      default
+Labels:         name=couchbase-operator
+Replicas:       1 desired | 1 updated | 1 total | 1 available | 0 unavailable
+Pod Template:
+  Labels:   name=couchbase-operator
+  Containers:
+   couchbase-operator:
+    Image:  couchbase/couchbase-operator:v1
+    Port:   8080/TCP
+    ...
    +
    +
    +
    +

    Notice that the deployment resource manages a ReplicaSet, which means +it’s important to check that the desired number of replicas match the +total number of available replicas when describing the Operator +deployment.

    +
    +
    +

    Additional information about the ReplicaSets and pods created by the +Operator deployment can be described using the deployment name as a +label selector. On Kubernetes:

    +
    +
    +
    +
    # replicasets created by couchbase-operator deployment
+$ kubectl describe rs -lname=couchbase-operator
+
+# pods created by the deployments replicaset
+$ kubectl describe pods -lname=couchbase-operator
+Name:       couchbase-operator-1917615544-8kgw9
+Status:     Running
+IP:     172.17.0.3
+Created By: ReplicaSet/couchbase-operator-1917615544
+Containers:
+  couchbase-operator:
+    Image:      couchbase/couchbase-operator:v1
+    ...
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    # replicasets created by couchbase-operator deployment
+$ oc describe rs -lname=couchbase-operator
+
+# pods created by the deployments replicaset
+$ oc describe pods -lname=couchbase-operator
+Name:       couchbase-operator-1917615544-8kgw9
+Status:     Running
+IP:     172.17.0.3
+Created By: ReplicaSet/couchbase-operator-1917615544
+Containers:
+  couchbase-operator:
+    Image:      couchbase/couchbase-operator:v1
+    ...
    +
    +
    +
    +
    +

    Server Pods

    +
    +

    Couchbase pods can be listed using the -lapp=couchbase label.

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    $ kubectl get po -l app=couchbase
+NAME                  READY     STATUS    RESTARTS   AGE
+cb-development-0000   1/1       Running   0          1h
+cb-development-0001   1/1       Running   0          1h
+cb-production-0000    1/1       Running   0          1h
+cb-production-0001    1/1       Running   0          1h
    +
    +
    +
    +

    On OpenShift, use the following command:

    +
    +
    +
    +
    $ oc get po -l app=couchbase
    +
    +
    +
    +

    The pods are also labeled by cluster and according to Couchbase service, +which makes it possible to get only the pods providing a specific +service for a specific cluster. For example, the following command gets +only the pods providing the query service on the cb-development +cluster. On Kubernetes:

    +
    +
    +
    +
    # query pods on dev deployment
+$ kubectl get po -l couchbase_cluster=cb-development,couchbase_service_query
+NAME                  READY     STATUS    RESTARTS   AGE
+cb-development-0000   1/1       Running   0          1h
+cb-development-0001   1/1       Running   0          1h
    +
    +
    +
    +

    On OpenShift, use the following command:

    +
    +
    +
    +
    # query pods on dev deployment
+$ oc get po -l couchbase_cluster=cb-development,couchbase_service_query
    +
    +
    +
    +

    After listing the pods, additional information can be collecting using +kubectl describe on Kubernetes as follows:

    +
    +
    +
    +
    $ kubectl describe po cb-development-0000
+Name:       cb-development-0000
+Namespace:  default
+Node:       minikube/192.168.99.100
+Start Time: Tue, 23 Jan 2018 07:49:38 -0700
+Labels:     app=couchbase
+        couchbase_cluster=cb-development
+...
+Containers:
+  couchbase-server:
+    Container ID:   docker://6b7aeb66cb540c6904b8e514776c180f7fbbea25656e6175972c408042db5713
+    Image:      couchbase/server:enterprise-5.0.1
    +
    +
    +
    +

    On OpenShift, use the following command:

    +
    +
    +
    +
    $ oc describe po cb-development-0000
    +
    +
    +
    +
    +

    Services

    +
    +

    Services are created to facilitate both pod to pod communication and +connections from external clients to the internal cluster. The former is +established using a headless ClusterIP service, and the latter via the +NodePort service. You can read more about the Kubernetes services +here. +When listing the services, the name of the ClusterIP service will match +the name provided in the CouchbaseCluster spec. The NodePort service is +also named after the CouchbaseCluster name, except with a `-ui' suffix.

    +
    +
    +

    Services belonging to the Operator can be listed as follows: On +Kubernetes:

    +
    +
    +
    +
    # all services managed by the operator
+$ kubectl get service -lapp=couchbase
+NAME                CLUSTER-IP   EXTERNAL-IP   PORT(S)                          AGE
+cb-development      None         <none>        8091/TCP,18091/TCP               2m
+cb-development-ui   10.0.0.7     <nodes>       8091:31822/TCP,18091:30155/TCP   2m
+cb-production       None         <none>        8091/TCP,18091/TCP               1m
+cb-production-ui    10.0.0.216   <nodes>       8091:30139/TCP,18091:30461/TCP   1m
+
+# only services used by cb-production cluster
+$ kubectl get service -lcouchbase_cluster=cb-production
+NAME               CLUSTER-IP   EXTERNAL-IP   PORT(S)                          AGE
+cb-production      None         <none>        8091/TCP,18091/TCP               2m
+cb-production-ui   10.0.0.216   <nodes>       8091:30139/TCP,18091:30461/TCP   2m
    +
    +
    +
    +

    On OpenShift, use the following commands:

    +
    +
    +
    +
    # all services managed by the operator
+$ oc get service -lapp=couchbase
+NAME                CLUSTER-IP   EXTERNAL-IP   PORT(S)                          AGE
+cb-development      None         <none>        8091/TCP,18091/TCP               2m
+cb-development-ui   10.0.0.7     <nodes>       8091:31822/TCP,18091:30155/TCP   2m
+cb-production       None         <none>        8091/TCP,18091/TCP               1m
+cb-production-ui    10.0.0.216   <nodes>       8091:30139/TCP,18091:30461/TCP   1m
+
+# only services used by cb-production cluster
+$ oc get service -lcouchbase_cluster=cb-production
+NAME               CLUSTER-IP   EXTERNAL-IP   PORT(S)                          AGE
+cb-production      None         <none>        8091/TCP,18091/TCP               2m
+cb-production-ui   10.0.0.216   <nodes>       8091:30139/TCP,18091:30461/TCP   2m
    +
    +
    +
    +

    Describing services

    +
    +

    After listing the services, additional information about ports and +endpoints can be gathered on Kubernetes using kubectl describe as +follows:

    +
    +
    +
    +
    $ kubectl describe service cb-production
+
+Name:           cb-production
+Namespace:      default
+Labels:         app=couchbase
+            couchbase_cluster=cb-production
+Annotations:        service.alpha.kubernetes.io/tolerate-unready-endpoints=true
+Selector:       app=couchbase,couchbase_cluster=cb-production
+Type:           ClusterIP
+IP:         None
+Port:           cb-admin    8091/TCP
+Endpoints:      172.17.0.10:8091,172.17.0.9:8091
+Port:           cb-admin-ssl    18091/TCP
+Endpoints:      172.17.0.10:18091,172.17.0.9:18091
+Session Affinity:   None
+Events:         <none>
    +
    +
    +
    +

    On OpenShift, use the following command:

    +
    +
    +
    +
    $ oc describe service cb-production
    +
    +
    +
    +

    You should expect that all pods within a cluster also exist as an +endpoint to each service.

    +
    +
    +
    +
    +

    Listing Custom Resources

    +
    +

    Custom resources are extensions of the Kubernetes API. The Couchbase +Operator creates custom resources of type CouchbaseCluster for each +cluster being deployed. Custom resources can be listed and described +using kubectl just as other built-in resources. On Kubernetes:

    +
    +
    +
    +
    $ kubectl get couchbasecluster
+NAME             KIND
+cb-development   CouchbaseCluster.v1beta1.couchbase.database.couchbase.com
+cb-production    CouchbaseCluster.v1beta1.couchbase.database.couchbase.com
+
+$ kubectl describe couchbasecluster cb-production
+Name:       cb-production
+Namespace:  default
+Labels:     <none>
+Annotations:    kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"couchbase.database.couchbase.com/v1beta1","kind":"CouchbaseCluster","metadata":{"annotations":{},"name":"cb-production","namespace":"def...
+API Version:    couchbase.database.couchbase.com/v1beta1
+Kind:       CouchbaseCluster
+...
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    $ oc get couchbasecluster
+NAME             KIND
+cb-development   CouchbaseCluster.v1beta1.couchbase.database.couchbase.com
+cb-production    CouchbaseCluster.v1beta1.couchbase.database.couchbase.com
+
+$ oc describe couchbasecluster cb-production
+Name:       cb-production
+Namespace:  default
+Labels:     <none>
+Annotations:    kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"couchbase.database.couchbase.com/v1beta1","kind":"CouchbaseCluster","metadata":{"annotations":{},"name":"cb-production","namespace":"def...
+API Version:    couchbase.database.couchbase.com/v1beta1
+Kind:       CouchbaseCluster
+...
    +
    +
    +
    +

    Describing the cluster resource returns the deployed specification and +its active status. For more information about the response returned from +the describe command, see Cluster Status +Guide.

    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/logsTroubleshooting.html b/content/operator/logsTroubleshooting.html new file mode 100644 index 0000000000..f419f7cd13 --- /dev/null +++ b/content/operator/logsTroubleshooting.html @@ -0,0 +1,725 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Logs and Troubleshooting

    +
    +
    +

    This section provides information about how to diagnose and troubleshoot +problems with the Couchbase Operator or your deployment.

    +
    +
    +

    When troubleshooting the Couchbase Operator, it is important to rule out +Kubernetes itself as the root cause of the problem you are experiencing. +See the Kubernetes +Troubleshooting +Guide for information about debugging applications within a Kubernetes +cluster.

    +
    +
    +

    The following topics are also helpful to understand when troubleshooting +the Operator:

    +
    +
    + +
    +
    +

    Full Deployment Logs

    +
    +

    To retrieve the logs for the Operator and the entire Kubernetes +deployment (nodes, pods, services…), run the following script: +cb_k8s_support.sh

    +
    +
    +
    +
    wget https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-kubernetes-support.sh
+chmod +x couchbase-kubernetes-support.sh
+./couchbase-kubernetes-support.sh
    +
    +
    +
    +

    The script gathers information about your Kubernetes deployment and +creates an archive within your current working directory:

    +
    +
    +
    +
    <cwd>/cb-k8s-support-01182018-14_00_18.tgz
    +
    +
    +
    +

    The script runs kubectl top pod to gather pod metrics such as CPU and +memory. By default, these metrics are empty unless you have heapster +deployed alongside your cluster. Run the following commands to deploy +heapster within your Kubernetes environment:

    +
    +
    +
    +
    git clone https://github.com/kubernetes/heapster.git
+cd heapster
+kubectl create -f deploy/kube-config/influxdb/
+kubectl create -f deploy/kube-config/rbac/heapster-rbac.yaml
    +
    +
    +
    +

    For OpenShift, refer to the OpenShift installation documentation about +cluster +metrics.

    +
    +
    +

    Run the script again to generate new logs that include pod metrics.

    +
    +
    +
    +

    Operator Logs

    +
    +

    The Couchbase Operator generates logs that can help troubleshoot your +deployment. Using kubectl or oc, you can choose to print the +Operator logs to stdout.

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    # Get name of operator pod
+$ kubectl get po -lapp=couchbase-operator
+NAME                                  READY     STATUS    RESTARTS   AGE
+couchbase-operator-1917615544-h20bm   1/1       Running   0          20h
+
+# Get logs
+$ kubectl logs couchbase-operator-1917615544-h20bm
+time="2018-01-23T22:56:34Z" level=info msg="Obtaining resource lock" module=main
+time="2018-01-23T22:56:34Z" level=info msg="Starting event recorder" module=main
+time="2018-01-23T22:56:34Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main
+time="2018-01-23T22:56:51Z" level=info msg="I'm the leader, attempt to start the operator" module=main
+time="2018-01-23T22:56:51Z" level=info msg="Creating the couchbase-operator controller" module=main
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    # Get name of operator pod
+$ oc get po -lapp=couchbase-operator
    +
    +
    +
    +

    Watch for the following messages which indicate that the Operator is +unable to reconcile your cluster into a desired state:

    +
    +
    +
      +
    • +

      Logs with level=error

      +
      • +
    • +

      Operator is unable to get cluster state after N retries

      +
      • +
    +
    +
    +
    +

    Getting Couchbase Server Logs

    +
    +

    The easiest way to get cbcollect logs is to use the standard logs +collection feature in the Couchbase Server Web Console. Go to Logs > +Collect Information, select the desired nodes, and click Collect +Logs.

    +
    +
    +

    You can also deploy a job within Kubernetes to trigger log collection. +The following commands collect the logs from all the nodes in the +cluster. The logs (when the default configuration is used) are found +under /opt/couchbase/var/lib/couchbase/tmp location.

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    kubectl create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cli-collect-logs.yaml
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    oc create -f https://packages.couchbase.com/kubernetes/0.8.1-beta2/couchbase-cli-collect-logs.yaml
    +
    +
    +
    +

    Note: If you are not using the default namespace, you must download +and update the couchbase-cli-collect-logs.yaml file to reflect your +namespace. For example, if your namespace is myproject, edit the +command field in the YAML file to replace +cb-example-0000.cb-example.default.svc with +cb-example-0000.cb-example.myproject.svc. The updated field will now +look like the following:

    +
    +
    +
    +
    ---
+command: [""/bin/sh"", ""-c"", ""couchbase-cli-secure collect-logs-start
+                              -c cb-example-0000.cb-example.myproject.svc
+                              -u {auth.admin.username}
+                              -p {auth.admin.password}
+                              --all-nodes""]
+---
    +
    +
    +
    +

    Once the log collection is complete, you can view the log location for +each node from the Couchbase Server Web Console by going to Logs > +Collect Information and clicking Show Current Collection. You can +then run a command like the following for each node in the cluster to +collect their logs.

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    kubectl cp <namespace>/<pod_name>:<path_to_logs> -c couchbase-server ./logs.zip
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    oc cp <namespace>/<pod_name>:<path_to_logs> -c couchbase-server ./logs.zip
    +
    +
    +
    +

    Here is an example command to collect the logs for node +cb-example-0000.

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    kubectl cp default/cb-example-0000:/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2017-09-28T175135-ns_1@127.0.0.1.zip -c couchbase-server ./logs.zip
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
        oc cp default/cb-example-0000:/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2017-09-28T175135-ns_1@127.0.0.1.zip -c couchbase-server ./logs.zip
    +
    +
    +
    +
    +

    See Also

    +
    +

    Refer to the +Couchbase +Server Troubleshooting guide for additional information about reporting +issues.

    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/nodeRecovery.html b/content/operator/nodeRecovery.html new file mode 100644 index 0000000000..831f027778 --- /dev/null +++ b/content/operator/nodeRecovery.html @@ -0,0 +1,530 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Node Recovery

    +
    +
    +

    The Couchbase Operator can detect node failures, rebalance out bad +nodes, and bring the cluster back up to the desired capacity. In most +cases, all of this happens automatically without any user intervention. +In a typical case where a node becomes unresponsive, the Operator waits +for Couchbase Server to recognize the node as being down and fail it +over. Couchbase Server waits for the node to be down for a specified +amount of time before failing over the node, and this failover timeout +can be set using the autoFailoverTimeout in the cluster configuration +file.

    +
    +
    +

    Once the node has been failed over, the Operator detects that the node +is down, marks it for removal, creates a new pod running Couchbase, and +finally rebalances the cluster. By removing the faulty node from the +cluster and adding a new node to the cluster, the Operator ensures that +the cluster is back up to the desired configuration.

    +
    +
    +

    What if multiple nodes fail?

    +
    +

    Currently, the Couchbase Operator can only failover one node at a time +in the cluster during auto-failover. This means that if multiple nodes +fail, some manual intervention is needed. In this case, you are required +to manually failover all the nodes that are down. The Operator will then +take care of adding new nodes back into the cluster. The easiest way to +failover downed nodes is to log into the Couchbase Web Console and click +the failover button for each downed node.

    +
    +
    +

    After the nodes have been failed over, you must check to see if the +failures resulted in any loss of data. If data was lost, then take +appropriate steps to recover the data by restoring from a backup or +setting up an XDCR replication from a remote cluster.

    +
    +
    +
    +

    Why not use a Persistent Volume and move the data to another pod?

    +
    +

    Currently, the Couchbase Operator does not support the use of Persistent +Volumes.

    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/operator.ditamap b/content/operator/operator.ditamap new file mode 100644 index 0000000000..9d578e4457 --- /dev/null +++ b/content/operator/operator.ditamap @@ -0,0 +1,42 @@ + + + + Couchbase Operator + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/operator/operatorConfig.html b/content/operator/operatorConfig.html new file mode 100644 index 0000000000..6979d7b45a --- /dev/null +++ b/content/operator/operatorConfig.html @@ -0,0 +1,580 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Couchbase Operator Configuration

    +
    +
    +

    The Couchbase Operator configuration is defined below. When loaded into +Kubernetes, it downloads the Couchbase Operator Docker image, creates +the CouchbaseCluster Custom Resource Definition (CRD), and starts +listening for CouchbaseCluster events. The Operator is defined as a +deployment to allow it to be restarted in the event of a pod or node +failure.

    +
    +
    +
    +
    apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: couchbase-operator
+  namespace: default
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        name: couchbase-operator
+    spec:
+      containers:
+      - name: couchbase-operator
+        image: couchbase/k8s-operator:0.8.1-beta
+        command:
+        - couchbase-operator
+        args:
+        - -create-crd
+        env:
+        - name: MY_POD_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: MY_POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        ports:
+          - name: readiness-port
+            containerPort: 8080
+        readinessProbe:
+          httpGet:
+            path: /readyz
+            port: readiness-port
+          initialDelaySeconds: 3
+          periodSeconds: 3
+          failureThreshold: 19
+        serviceAccountName: couchbase-operator
    +
    +
    +
    +

    Most of the fields in the Operator configuration should never be changed +and we recommend using the configuration as is. However, there are some +exceptions noted below.

    +
    +
    +

    Changing the Namespace

    +
    +

    The Operator manages clusters in the namespace that it is deployed in. +If you want to deploy the Operator in a namespace other than the default +namespace, then change the metadata.namespace field.

    +
    +
    +
    +

    Changing the Operator Container Image

    +
    +

    You should not need to change the Operator image unless you are pulling +the image from somewhere other than the official Couchbase Docker +repository. If you are pulling the image from somewhere else, change the +spec.spec.containers[0].image field.

    +
    +
    +
    +

    Changing the Name

    +
    +

    By default the name of the deployment created to maintain the Couchbase +Operator is called couchbase-operator. We recommend keeping this name +since it is used in all of our examples and tutorials. If you need to +change it for some reason, ensure that you change the metadata.name, +spec.template.metadata.labels.name, and spec.spec.containers[0].name +fields. These fields also must all have the same value.

    +
    +
    +
    +

    Changing the Service Account

    +
    +

    We recommend using a ServiceAccount named couchbase-operator, but +depending on your environment you may want to use a service account with +a different name. Note that this field only takes effect if your +Kubernetes environment has RBAC enabled.

    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/overview.html b/content/operator/overview.html new file mode 100644 index 0000000000..4309e3e904 --- /dev/null +++ b/content/operator/overview.html @@ -0,0 +1,573 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Overview

    +
    +
    +

    The Couchbase Operator is a Kubernetes native integration that enables +you to automate the management of common Couchbase tasks such as the +configuration, creation, and scaling of Couchbase clusters. By reducing +the complexity of running a Couchbase cluster, it lets you focus on the +desired configuration and not worry about the details of manual +deployment and lifecycle management.

    +
    +
    +

    How It Works

    +
    +

    The Couchbase Operator extends the Kubernetes API by creating a Custom +Resource Definition(CRD) and registering a Couchbase specific +controller(the Operator) to manage Couchbase clusters. The CRD allows +you to define a configuration describing what a Couchbase cluster should +look like. For example, a configuration might define a cluster with +three nodes, one bucket, and 8GB of memory for the data service. Once +the configuration is loaded into Kubernetes, the configuration is passed +to the custom Couchbase controller which takes actions to ensure a +Couchbase cluster with the specified configuration is provisioned. The +controller can also detect updates to the configuration and reacts to +changes that occur in the cluster itself. Like all Kubernetes standard +built-in resources, the Couchbase Operator doesn’t just manage a single +Couchbase cluster but multiple Couchbase clusters across an entire +Kubernetes deployment.

    +
    +
    +
    +

    Why Not Just Use StatefulSets

    +
    +

    StatefulSets are great for certain use cases, but they don’t work that +well when running complex software like databases. This is because +StatefulSets focus on creating and managing pods, not on managing the +software running on them. For example, if you wanted a 4-nodes cluster +and deployed Couchbase using StatefulSets, you would get 4 uninitialized +Couchbase pods that don’t know about each other. It would then be up to +you to cluster the nodes together, and this means extra operational +tasks.

    +
    +
    +

    By creating our own custom Couchbase controller, we can add Couchbase +specific knowledge so that as each Couchbase pod is deployed we can +properly configure it and join it with the other Couchbase pods in the +cluster. It’s also important to keep in mind that provisioning a cluster +is just one place where having a custom controller helps us to automate +tasks. Node failure, ad-hoc scaling, and many other management tasks +require Couchbase specific knowledge to manage Couchbase clusters.

    +
    +
    +
    +

    What It Supports

    +
    +

    The goal of the Couchbase Operator is to fully manage one or more +Couchbase deployments so that you as a developer/SRE don’t need to worry +about the operational complexities of running Couchbase. This is, +however, a beta release, so not all features have been added yet. Below +is a list of the management tasks we currently support:

    +
    +
    +
      +
    • +

      Cluster configuration

      +
      • +
    • +

      Cluster provisioning

      +
      • +
    • +

      Ad-hoc scalability

      +
      • +
    • +

      Auto recovery

      +
      • +
    • +

      Logging

      +
      • +
    +
    +
    +
    +

    Get Started

    +
    +

    Prerequisites and Setup

    +
    +
    +

    Release Notes

    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/prerequisiteAndSetup.html b/content/operator/prerequisiteAndSetup.html new file mode 100644 index 0000000000..200f9b4bfe --- /dev/null +++ b/content/operator/prerequisiteAndSetup.html @@ -0,0 +1,548 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Prerequisites and Setup

    +
    +
    +

    In order to run the Couchbase Operator all you need is a running +Kubernetes or OpenShift cluster. We support the following releases:

    +
    +
    +
      +
    • +

      Kubernetes 1.8+

      +
      • +
    • +

      OpenShift 3.9+

      +
      • +
    +
    +
    +

    If you do not have access to a Kubernetes cluster and plan on using the +Couchbase Operator for development, we recommend using either MiniKube +(single-node Kubernetes cluster) or MiniShift (single-node OpenShift +cluster). Both these products are much easier to install and deploy when +compared to setting up and running an actual Kubernetes or OpenShift +cluster. For installation instructions, see +MiniKube or +MiniShift.

    +
    +
    +

    Preparing the cluster to run the Couchbase Operator may require setting +up proper RBAC settings in your Kubernetes cluster. Before moving +forward, you should read the deployment guides for for the various +supported platforms.

    +
    +
    + +
    +
    +

    For more information about how the Couchbase Operator Deployment is +defined, as well as documentation about the fields that you might want +to customize, see the Couchbase Operator +Configuration.

    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/releaseNotes.html b/content/operator/releaseNotes.html new file mode 100644 index 0000000000..52af149e88 --- /dev/null +++ b/content/operator/releaseNotes.html @@ -0,0 +1,714 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    v0.8.1 [Beta 2] Update

    +
    +
    +

    The v0.8.1 build of the Couchbase Operator resolves a version check issue that prevented the Operator from initializing after deployment. An error similar to the following could be seen in the Kubernetes log:

    +
    +
    +
    +
    "Initialization failed: kubernetes version 1.0 is too old, 1.8 is minimum supported version "
    +
    +
    +
    +

    If you encountered this issue, please reinstall the Operator using the latest documentation, as the commands have been updated to reflect the latest version number.

    +
    +

    v0.8.0 [Beta 2] Release Notes

    +
    +
    +

    The Couchbase Operator is a Kubernetes native integration that enables +you to automate the management of common Couchbase tasks such as the +configuration, creation, and scaling of Couchbase clusters. The +Couchbase Operator 0.8.0 [Beta 2] was released in May 2018.

    +
    +
    +

    We consider beta releases to have some rough edges and bugs and hence +are intended for development purposes only. Since the release is still +under active development, you can have a big impact on the final version +of the product and its documentation by providing feedback and +observations. We encourage you to post your questions and feedback to +the Couchbase +Forums.

    +
    +
    +

    Limitations

    +
    +

    The Beta 2 release has the following limitations:

    +
    +
    +
      +
    • +

      CRD validation is supported through cbopctl and partial CRD validation +is supported though standard Kubernetes/OpenShift tools such as +kubectl/oc. We recommend using cbopctl when submitting CouchbaseCluster +configurations to Kubernetes/OpenShift.

      +
      • +
    • +

      The Couchbase Operator can only failover one node at a time in the +cluster during auto-failover. This means that if multiple nodes fail, +some manual intervention is needed.

      +
      • +
    • +

      The Couchbase Operator does not support the use of Persistent Volumes.

      +
      • +
    +
    +
    +
    +

    Known Issues

    +
    + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Table 1. Known Issues
    IssueDescription

    K8S-155

    The Couchbase Operator logs show error messages while the cluster is being built. These are harmless +and do not impact the cluster creation.

    K8S-119

    If a cluster is scaled up beyond the available memory, rebalance fails and the cluster becomes unavailable.

    K8S-112

    A rebalance started event is posted after the actual rebalance operation is completed. And no event is posted to indicate that the +operation has completed.

    K8S-80

    The Couchbase Web Console may redirect to a node even after it has been removed from the cluster.

    K8S-195

    The OpenShift bootstrap script has issues running in OpenShift clusters, but works in MiniShift. Note that we don’t recommend using +this bootstrap script in production clusters.

    K8S-205

    Sometimes an extra bucket event will show up in the events log if a node is manually failed over.

    +
    +
    +

    Fixed Issues

    + + ++++ + + + + + + + + + + + + + + + + + + + + + + + + +
    Table 2. Fixed Issues
    IssueDescription

    K8S-172

    The index storage mode cannot be changed.

    K8S-169

    There is no CRD validation, so the Couchbase configurations uploaded to Kubernetes are not validated. An incorrect configuration can +cause all kinds of problems, including potentially crashing the Operator.

    K8S-155

    The Couchbase Operator logs show error messages while the cluster is being built. These are harmless and do not impact the cluster +creation.

    K8S-92

    Changing the cluster-name field in the example/couchbase-cluster.yaml file causes the cluster get into an +inconsistent state.

    +
    + + +
    +

    v0.7 [Beta] Release Notes

    +
    +
    +

    The Couchbase Operator is a Kubernetes native integration that enables +you to automate the management of common Couchbase tasks such as the +configuration, creation, and scaling of Couchbase clusters. The +Couchbase Operator 0.7 [Beta] was released in February 2018.

    +
    +
    +

    We consider beta releases to have some rough edges and bugs and hence +are intended for development purposes only. Since the release is still +under active development, you can have a big impact on the final version +of the product and its documentation by providing feedback and +observations.

    +
    +
    +

    Limitations

    +
    +

    The Beta release has the following limitations:

    +
    +
    +
      +
    • +

      There is no CRD validation at this time, so the Couchbase +configurations uploaded to Kubernetes are not verified by the Couchbase +Operator.

      +
      • +
    • +

      The Couchbase Operator can only failover one node at a time in the +cluster during auto-failover. This means that if multiple nodes fail, +some manual intervention is needed.

      +
      • +
    • +

      The Couchbase Operator does not support the use of Persistent Volumes.

      +
      • +
    +
    +
    +
    +

    Known Issues

    + + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Table 3. Known Issues
    IssueDescription

    K8S-181

    When running on an OpenShift cluster, pod metrics fail to be generated.

    K8S-172

    The index storage mode cannot be changed.

    K8S-169

    There is no CRD validation, so the Couchbase configurations uploaded to Kubernetes are not validated. An incorrect configuration can +cause all kinds of problems, including potentially crashing the Operator.

    K8S-155

    The Couchbase Operator logs show error messages while the cluster is being built. These are harmless and do not impact the cluster +creation.

    K8S-119

    If a cluster is scaled up beyond the available memory, rebalance fails and the cluster becomes unavailable.

    K8S-112

    A rebalance started event is posted after the actual rebalance operation is completed. And no event is posted to indicate that the +operation has completed.

    K8S-92

    Changing the cluster-name field in the example/couchbase-cluster.yaml file causes the cluster get into an inconsistent state.

    K8S-80

    The Couchbase Web Console continues to display a node even after it has been removed from the cluster.

    +
    +
    +
    +
    + + + + +
    + + + + + + + + + + + + + diff --git a/content/operator/runningCouchbase.html b/content/operator/runningCouchbase.html new file mode 100644 index 0000000000..201f7c3ca9 --- /dev/null +++ b/content/operator/runningCouchbase.html @@ -0,0 +1,505 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Running Couchbase on Kubernetes or OpenShift

    +
    +
    +

    After deploying a Couchbase cluster using a CRD (deployment blueprint), +the Couchbase Operator constantly watches the deployment blueprint for +changes and analyzes the differences. Based on the updated information, +the Couchbase Operator modifies the Couchbase cluster to reflect the +changes in the deployment configuration. +How Couchbase Operator Works

    +
    +
    +

    Using a deployment blueprint makes it easy to scale clusters and +services up and down. The section on Scaling a +CouchbaseCluster describes how to scale up or scale down a +CouchbaseCluster using the deployment blueprint.

    +
    +
    +

    A deployment blueprint also helps detect node failures, rebalance out +bad nodes, and bring the cluster back to the desired capacity. The +section on Node Recovery provides further details.

    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/scalingCouchbase.html b/content/operator/scalingCouchbase.html new file mode 100644 index 0000000000..6a6287099d --- /dev/null +++ b/content/operator/scalingCouchbase.html @@ -0,0 +1,624 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Scaling a CouchbaseCluster

    +
    +
    +

    Using the Couchbase Operator, you can easily scale clusters and services +up and down. The handling of individual nodes is defined in the servers +section of the Couchbase cluster +specification. Here is an example of a definition for a Couchbase +cluster where all nodes run the data, index, search, and query services.

    +
    +
    +
    +
    servers:
+  - size: 3
+    name: all_services
+    services:
+      - data
+      - index
+      - query
+      - search
+    dataPath: /opt/couchbase/var/lib/couchbase/data
+    indexPath: /opt/couchbase/var/lib/couchbase/data
    +
    +
    +
    +

    This configuration specifies that the cluster should contain 3 nodes +running the data, index, query, and search services and that the data +and index path are each set to /opt/couchbase/var/lib/couchbase/data. +To scale these servers, you only need to change the size parameter and +then push the new configuration into Kubernetes to increase or decrease +the number of nodes in the cluster. To update the configuration in +Kubernetes, run the following command:

    +
    +
    +

    On Kubernetes:

    +
    +
    +
    +
    kubectl replace -f <path to config>
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    oc replace -f <path to config>
    +
    +
    +
    +

    Alternatively you can also edit the configuration of a Couchbase cluster +directly by running the command below. On Kubernetes:

    +
    +
    +
    +
    kubectl edit cbc <name of cluster>
    +
    +
    +
    +

    On OpenShift:

    +
    +
    +
    +
    oc edit cbc <name of cluster>
    +
    +
    +
    +

    This command pulls the current configuration for the cluster from +Kubernetes and opens it up in an editor. You can then edit the +configuration, and upon saving and closing the file, the updated +configuration is automatically pushed back into Kubernetes.

    +
    +
    +

    One thing you will notice when scaling a cluster is that the servers +section takes a list of server configurations. The configuration is +setup this way so that services can be scaled independently. For +example, if your Couchbase cluster contains 3 nodes running the data +service, one node running the query service, and one node running the +index service, then the servers section of your configuration would look +like this:

    +
    +
    +
    +
    servers:
+  - size: 3
+    name: data_services
+    services:
+      - data
+    dataPath: /opt/couchbase/var/lib/couchbase/data
+    indexPath: /opt/couchbase/var/lib/couchbase/data
+  - size: 1
+    name: index_services
+    services:
+      - index
+    dataPath: /opt/couchbase/var/lib/couchbase/data
+    indexPath: /opt/couchbase/var/lib/couchbase/data
+  - size: 1
+    name: query_services
+    services:
+      - query
+    dataPath: /opt/couchbase/var/lib/couchbase/data
+    indexPath: /opt/couchbase/var/lib/couchbase/data
    +
    +
    +
    +

    This configuration creates a 5-node Couchbase cluster with different +services running on each pod. If you see that your queries are taking +longer than usual, you can scale up your query service independent of +the other services by changing the size of the server specification that +contains the query service. For example, if you update the size from 1 +to 2 and push the new configuration into Kubernetes, the Couchbase +Operator will automatically scale up the query service.

    +
    +
    +

    The serverName

    +
    +

    The serverName in the servers section is a very important parameter +that is used to track a group of similarly deployed pods. This name is +used internally by the operator to track servers of a given type. +Changing this name will cause pods that have been deployed in the server +group to be removed from the cluster and the operator will deploy new +pods tagged with the new serverName even if no other parameters in the +server group have been changed. As a result, we recommend that users +never change the serverName for a particular server group except in +the scenario defined below.

    +
    +
    +
    Changing a Provisioned Node’s Services
    +
    +

    If you’ve used Couchbase Server in the past then you have likely found +that once a Couchbase Server node is joined to a cluster, the services +running on that node cannot change. The Couchbase Operator allows users +to edit the configuration in a way that allows easily changing the +services by automating the removal of the old nodes and adding in the +new nodes with the desired services. To do this, you can change both the +serverName and services parameters during the same configuration +update.

    +
    +
    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/operator/updatingCouchbase.html b/content/operator/updatingCouchbase.html new file mode 100644 index 0000000000..9c44300fe4 --- /dev/null +++ b/content/operator/updatingCouchbase.html @@ -0,0 +1,527 @@ + + + + + + Couchbase Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Couchbase + + + + + + + + + + + + +
    +
    + +
    +
    + +
    + + +
    +
    +

    Updating a CouchbaseCluster

    +
    +
    +

    Changes to a CouchbaseCluster should never be done manually as the +Operator will revert the changes if they do not match what is in the +configuration. Always update the cluster configuration using the +kubectl command. We recommend using either the kubectl replace or +kubectl edit command. If you’re using OpenShift, use the oc replace +or oc edit command instead.

    +
    +
    +

    To use the kubectl replace command, first edit a field in the +Couchbase cluster configuration file stored on your local disk, and then +run the following command:

    +
    +
    +
    +
    kubectl replace -f <path to updated config>
    +
    +
    +
    +

    Using the kubectl edit command you can edit the Couchbase cluster +configuration directly. For example, if your cluster name is +cb-example, use the following command to edit the configuration:

    +
    +
    +
    +
    kubectl edit cbc cb-example
    +
    +
    +
    +

    This opens a text editor with the current configuration for the +cb-example cluster. You can then edit the configuration, save, and +then close the text editor. Upon closing the text editor, the changes +will be pushed back to Kubernetes.

    +
    +
    +

    It is important to remember that some fields in the Couchbase cluster +configuration cannot be edited after the cluster is created. The +Couchbase Cluster Configuration +documentation notes such fields in their respective descriptions. The +current version does not support field validation.

    +
    +
    +
    +
    + + + +
    +
    +
    + + + + + + + + + + + + diff --git a/content/performance/index_pushdowns.dita b/content/performance/index_pushdowns.dita index baf235af29..3c3ccff5a0 100644 --- a/content/performance/index_pushdowns.dita +++ b/content/performance/index_pushdowns.dita @@ -7,7 +7,7 @@ possible performance for N1QL queries. The GSI Indexes are a vital part of query performance and are tightly coupled with the N1QL engine. The following links explain the basics of Indexing and how it helps query performance:
      -
    • Indexing
      • +
    • Indexes
    • Indexes for N1QL
    • Indexing JSON diff --git a/content/release-notes.ditamap b/content/release-notes.ditamap index 459656b829..982ded835d 100644 --- a/content/release-notes.ditamap +++ b/content/release-notes.ditamap @@ -4,6 +4,5 @@ Release Notes - diff --git a/content/release-notes/relnotes-v5.0.x.dita b/content/release-notes/relnotes-v5.0.x.dita new file mode 100644 index 0000000000..7cd3f8052c --- /dev/null +++ b/content/release-notes/relnotes-v5.0.x.dita @@ -0,0 +1,914 @@ + + + + Release Notes for Couchbase Server 5.0.x + +

      Couchbase Server 5.0 delivers exciting capabilities for cross datacenter replication, + security, query, search, tools, and application development. Take a look at What's new for a list of new features and + improvements that are available in this release.

      +
      Release 5.0.1 +

      Couchbase Server 5.0.1, released in December 2017, supersedes version 5.0.0 released + earlier this year. Version 5.0.1 is the first maintenance release in the 5.0.x series for + Couchbase Server.

      +

      Fixed Issues

      + +

      Cross Datacenter Replication (XDCR)

      + + + + + + + Issue + Description + + + + + MB-27162 + Revision-based conflict resolution did not work as expected as the revision + number was not updated for a create that immediately followed a delete operation. + The revision number of a deleted document was updated after the item was added to + the checkpoint queue, which meant that when you recreate that document, the revision + number was not being replicated intra-cluster or via XDCR. This has been fixed. + + + +
      + +

      Security

      + + + + + + + Issue + Description + + + + + MB-27203 + The "default" bucket is a bucket that in + versions prior to 5.0 was either suggested or automatically created for users during + the install and configuration process with a goal of being compatible with memcached + out of the box. As its intention was to allow users to quickly get started with + Couchbase it permitted anonymous access, consistent with memcached behavior. + Starting 5.0, all buckets require authenticated access.

      However, there was a bug in 5.0.0 + where under certain circumstances anonymous access could be allowed to the bucket. + This issue only affected a bucket with the name "default" and + could only be triggered by an authenticated user with administrative permissions. This + issue has been fixed.

      +       + + +
      +

      Tools

      + + + + + + + Issue + Description + + + + + MB-26556 + The couchbase-cli cluster-init command failed to + initialize all services in the Community Edition as the default storage mode + incorrectly pointed to Plasma, which is available in the Enterprise Edition, + instead of ForestDB. + + + +
      +

      +
      +
      Release 5.0.0 +

      Couchbase Server 5.0.0 was released in October 2017. This release includes new features and behavior changes. The following sections list the behavior changes, fixed issues, known issues, and deprecated items.

      +

      Major Behavior Changes Compared to Version 4.6

      +

      Here are the behavior changes in the product, compared to the previous (4.6.x) release.

      +

      +

        +
      • Newly created buckets in 5.0 (including sample buckets) do not have any passwords + associated with them. To access a bucket, a user must be created with appropriate roles + to access using RBAC. See Bucket Full Access for details.
        • +
      • Important: Many 5.0 features require the latest versions of Couchbase SDK. + It is critical to update the SDK before updating to 5.0. Upgrade according to the SDK + versions listed in the SDK Compatibility chart below.
        • +
      • The TAP protocol which was previously deprecated, is now removed. TAP is an internal + protocol that streams information about data changes between cluster nodes. TAP is + replaced with a new protocol called Database Change Protocol (DCP).
        • +
      • Starting 5.0, port based buckets can only be created now through the bucket CLI + command. Use the command to create a port based bucket. Server-side moxi which + drives port based bucket creation has been deprecated. Consider using client-side moxi + instead.
        • +
      • For Windows, .exe installers which were previously deprecated are + now removed and replaced with industry-standard MSI installer files. Consequently, any + upgrade for the Windows platform from pre-5.0 version to 5.0 will need to be done in a + rolling upgrade fashion. See for details.
        • +
      • Re-designed Couchbase Web + Console.
        • +
      +

      + +

      Known Issues and Limitations

      +

      The following table lists some of the known issues in this release.

      + + + + + + + Issue + Description + + + + + MB-27203 + Summary: The "default" bucket is a bucket that in + versions prior to 5.0 was either suggested or automatically created for users during + the install and configuration process with a goal of being compatible with memcached + out of the box. As its intention was to allow users to quickly get started with + Couchbase it permitted anonymous access, consistent with memcached behavior. + Starting 5.0, all buckets require authenticated access. Also in 5.0 there is a bug + where under certain circumstances anonymous access can be allowed to the bucket. + This issue only affects a bucket with the name "default" and + can only be triggered by an authenticated user with administrative permissions. This + issue will be fixed in the next maintenance release.

      Workaround: Do not + define a bucket named "default" in your cluster + configuration.

       +       + + MB-26171 + Summary: Creating two XDCR replications to the same cluster with + different users is not supported. + + + MB-26126 + Summary: When using Couchbase Server 5.0 with Sync Gateway 1.5+ and the + Mobile Convergence feature is enabled in Sync Gateway (enable_shared_bucket_access), + the count of non-resident documents reported in the UI may be incorrect. + + + MB-26085 + Summary: Rebalance failure is observed if the index planner takes a long + time ( > ns-server timeout) to execute. + + + MB-26065 + Summary: Rebalance exits when loading the memcached RBAC file. + + + MB-26037 + Summary: During a rebalance, if the destination node for a particular + vBucket already has a previous, outdated version of that vBucket then the rebalance + can initially fail.

      Workaround: Retry the rebalance; the second attempt + should succeed.

       +       + + MB-26022 + Summary: When a source topology changes, while the docs_processed and + changes_left stats are both recomputed, only the changes_left stats in the + statistics overview are updated. This leads to incorrect docs_processed and + incorrect, potentially negative changes_left statistics. + + + MB-25938 + Summary: Potential corruption of documents with extended + attributes(XATTRs) during high concurrency.

      If a document contains extended + attributes (XATTRs) and a Key/Value mutation request changes one of those XATTRs, + the previous XATTR values are written to in a non-atomic way. This means that + there is a potential race if another concurrent request was also accessing the + same document at the same time. The other request could read a partial or + corrupted value for the document.

      Note that this issue is believed to be very + rare and has only been seen under synthetic data race analysis. +

      Workaround: Do not modify XATTRs on documents which already have them. +

      +       + + MB-25935 + Summary: Data Service returns TEMP_FAIL for locked keys instead of + LOCKED.

      Data Service 5.0 adds extended error codes (XERROR) to provide richer + error handling between newer, smart SDKs and the server. As part of this, + attempting to mutate a document which is locked should return ERROR_LOCKED, + however it incorrectly returns TEMP_FAIL.

      Workaround: Check for both + LOCKED and TEMP_FAIL error codes when using pessimistic locking.       +       + + MB-25909 + Summary: When you upgrade the source node in a unidirectional + replication from 4.1 to 4.6, the GoGC global setting was set to 0 instead of a + higher number. + + + MB-25785 + Summary: Resuming a backup process on ephemeral buckets fails if there + was a data service rollback in the original backup that failed. This issue is + observed when there is a failover shortly after starting the first backup in a + backup repository. removing the original backup.

      Workaround: Remove the + original backup by using the --purge flag and then start the backup process again. +

      +       + + MB-25707 + Summary: UI errors in XDCR are observed even though XDCR continues to + function. + + + MB-23074 + Summary: Performance issues may be observed when running Couchbase + Server on CentOS 7.3 with kernel 3.10.0-514.6. + + + MB-22679 + Summary: The Full Text Search service doesn't delete pindex files when + deleting the index after a MOSS crash. + + + MB-21772 + Summary: Full Text Search queries fail under high load on multi-node + cluster due to a large number of connections. + + + MB-12000 + Summary: The rebalance progress reported during delta recovery is + confusing as certain phases like warmup are not tracked. + + + +
      +

      This section lists some of the limitations in different areas across the product:

      + + + + + + + Area + Description + + + + + Powerful Query, Indexing, and Search + +
        +
      • Index Replicas Placement

        When deciding the index nodes to + place index replicas, the system takes into account to not place replicas onto + the same node, and to distribute the replicas across as many server groups as + possible. It then chooses the nodes the have the fewest number of indexes to + place the replicas. In the future, we will improve the replica placement + algorithm to better optimize the memory, CPU, and disk usage across the index + nodes.

        • +
      • + Index Rebalance +

        When adding new indexer nodes to a cluster to increase the capacity + for indexes, the system will not move existing indexes onto the new nodes. To + rebalance indexes from existing nodes to the new nodes, user needs to eject + the existing nodes and add the new nodes. The system will then move indexes + from the ejected nodes to the new nodes while balancing resource utilization. + This is known as swap rebalance.

        Rebalance will not take into + consideration any explicit index placement specified when an index was + created. A user cannot perform create/drop/build index during index + rebalance.

        • +
      • + Online Index Upgrade +

        To perform online upgrade from version 4.x, you need to make sure + all indexes on an index have at least one equivalent index (index with same + definition) on other nodes. One then fails-over the index node, upgrades the + node to version 5.0, and then adds the node back to the cluster. While the + node is failed-over, the equivalent indexes on other nodes will be leveraged + for queries.

        Once the system has been upgraded to 5.0, you can + use swap rebalance to perform online upgrade to future versions.

        • +
      +       +       + + Upgrade + +

      If you are using GSI indexes with a previously GA'd release of Couchbase (such as + 4.x) or Couchbase 5.0 Beta versions, we suggest that you drop the older index + definitions and recreate them using 5.0 GA version with Plasma, the newly + introduced storage engine for GSI.

      +       +       + + N1QL Application Continuity:

      In 5.0 Beta 1 release, N1QL introduced multiple + performance enhancements enabled by a new internal protocol between Query and + Index services. While upgrading large Couchbase cluster deployments, the cluster + may be in a state where different Couchbase services are running on different + versions. In such scenarios, this feature ensures seamless continuity for N1QL + queries and applications irrespective of whether the query and indexing services + are running a 4.x or 5.0 version. The N1QL clients can avail the 5.0 features and + performance only when the issued query is processed by 5.0 query and indexing + services.

       +       + + Many 5.0 features require the latest versions of Couchbase SDK. It is critical + to update the SDK before updating to 5.0. Upgrade according to the SDK versions + listed in the SDK + Compatibility chart. + + + We’ve replaced the Windows .exe installers with + industry-standard MSI installer files in this release. Consequently, only rolling + upgrades from 4.x or earlier versions to 5.0 are supported. + + + SDK Compatibility + Many 5.0 features require you to upgrade your client SDK versions. The minimum + versions that support the latest 5.0 features are shown below, newer versions are available through the + Release Notes link. Regardless of needing + new features, it is always advised to upgrade to the newest SDK version.

      +

      + + + + + + + SDK + Min. Version for 5.0 + Reference + + + + + Java + 2.5.1 + Release notes + + + .NET + 2.5.0 + Release notes + + + Node.js + 2.4.0 + Release notes + + + Python + 2.2.6 + Release notes + + + PHP + 2.4.0 + Release notes + + + Go + 1.2.5 + Release notes + + + C + 2.8.0 + Release notes + + + +
      +

      + + + + + +

      Fixed Issues

      +

      This section lists some of the important fixed issues in this release.

      +

      Data Service

      + + + + + + + Issue + Description + + + + + MB-23976 + The TOUCH command modifies a document and updates a CAS value too. However, + the result of TOUCH incorrectly used a pre-updated CAS value and did not reflect + the updated CAS. + + + MB-21156 + As part of a number of improvements to logging, fixed an issue where + couchdb.log wrapped too quickly. + + + MB-20091 + Improvements to ForestDB's thread handling for improved compaction. + + + MB-16150 + Fixed inconsistency in curr_items with full reviction on Data Greater than + Memory (DGM). + + + MB-13087 + When an item is locked and there’s actually a lot of load on the server, the + server now returns a PROTOCOL_BINARY_RESPONSE_LOCKED response to the client + application if the client application has enabled extended error codes. + + + +
      +

      Full Text Search Service

      + + + + + + Issue + Description + + + + + MB-23271 + To improve readability, the stat keys in section mossScope are now sorted. + + + + MB-22560 + The FTS ports now adjust SSL cipher levels based on the + COUCHBASE_SSL_CIPHER_LIST environment variable. SSL ciphers can be set to LOW, + MEDIUM, or HIGH(default) levels. + + + MB-21855 + Improved error messages for the following scenarios:
        +
      • Creating an index with a name that already exists.
        • +
      • Creating an alias without target index.
        • +
      +       + + MB-20939 + Improved memory usage and resource containment for Full Text Indexes. + + + MB-18042 + We’ve removed the Byte Array Converter dropdown on the Advanced tab of the Full + Text Index editor as it contained a single value that couldn’t be changed.

      Note + that index definitions created in earlier releases will not work unless you remove + the "byte_array_converter": "json" attribute value pair from the index definition + JSON.

       +       + + +

      Indexing Service

      + + + + + + Issue + Description + + + + + MB-21594 + Duplicate partitions removed during rebalance to prevent OOM (Out Of Memory + error), and subsequent recovery problems. + + + MB-20297 + For an index that was built with defer_build:true, the CREATE INDEX statement + on the Indexes tab showed an invalid N1QL statement that could not be copied to run + as is using cbq or the Query Workbench. + + + +

      Installer

      + + + + + + Issue + Description + + + + + MB-22283 + Installing and running Couchbase Server as a non-root and non-sudo user is + now supported. + + + +
      + Query Service

      + + + + + + + Issue + Description + + + + + MB-25103 + Query Service authentication failed when the password contained a colon (":") + character. + + + MB-24429 + When fetching data, if the data node failed to respond due to some reason, + the query could hang as there was no timeout specified. This has been addressed by + passing the query request timeout parameter when fetching data. + + + MB-23150 + The monitoring view for running queries provides details such as the node + that’s processing the request, the client IP that initiated the request, and the + running program type (i.e. WorkBench, CBC, Java, .NET, CBQ, etc.). This helps + isolate rogue queries and unauthorized access. + + + MB-22831 + Inserting a numerical value using a N1QL query caused the value to be stored + in scientific notation. + + + MB-22273 + In Couchbase clusters with multiple query service nodes, applications may get + incorrect results from prepared N1QL queries that use covering indexes. + + + MB-22119 + When using ARRAY expressions, query service may rarely panic with error + referring to ‘slice allocations’. + + + MB-22105 + The UPDATE, INSERT, UPSERT statements may fail with the error message: + "Panic: : runtime error: slice bounds out of range", when running with increased + pipeline_batch settings. + + + MB-22093 + The query engine may intermittently throw error code 12008 because of a + memory allocation failure. + + + MB-21928 + The command line shell for N1QL, cbq, now connects to 8091 with http and + 18091 with https by default when the port is not specified. + + + MB-19893 + The SELECT RAW did not take ORDER BY into account, thus the results were not + ordered. + + + MB-18769 + Fixed an issue where a covered query with meta().type in the WHERE clause + gave in correct results. + + + + +
      +

      Security

      + + + + + + + Issue + Description + + + + + MB-24088 + The secrets management feature fails to encrypt secrets larger than 4KB (such + as SSL certificates). This causes the cluster manager to unexpectedly terminate + when saving the cluster configuration to disk. + + + MB-15624 + You can now specify a SSL cipher list on Windows platform. + + + +
      +

      Tools

      + + + + + + Issue + Description + + + + + MB-25669 + On Windows platform, the cbimport tool failed to import a tab delimited CSV + file where the tab is specified using '\t' or '\\t'. + + + MB-20403 + The cbbackupmgr merge command did not function correctly and caused data + corruption under the following circumstances:
        +
      • The bucket being backed up was flushed between incremental backups.
        • +
      • The bucket being backed up was deleted and re-created in between incremental + backups.
        • +
      • The bucket had not been backed up between a document's deletion and the + metadata for that deleted document being purged.
        • +
      • There had been a failover between incremental backups of the bucket.
        • +
      +       + + MB-10093 + Xmem XDCR network bandwidth can now be throttled if desired. + + + +
      + Web Console

      + + + + + + + Issue + Description + + + + + MB-23251 + The AutoComplete feature for the password field on the Web Console has been + disabled. + + + MB-12480 + The text displayed when changing the RAM quota for a bucket was misleading + and has been fixed to reflect the actual ranges that the server will allow the + bucket to be sized within. + + + MB-9658 + The Web Console document browser for a bucket displays the ID and content + sample of the documents in the bucket. The UI displayed only the first 16 + characters and affected usability when viewing documents with keys longer than 16 + characters. This has been fixed to display the full key name and wrap to the next + line if needed. + + + +
      +

      XDCR

      + + + + + + + Issue + Description + + + + + MB-22514 + The XDCR target topology change detection process resulted in a large number + of open connections which caused a high CPU usage. This has been fixed by reducing + the number of connections to the target bucket. + + + MB-22172 + If a mutation needs to be resent, sent_time is updated so that next mutation + can be sent at the appropriate interval. + + + MB-21369 + Fixed a replication connection leak caused by a failure to close connections + to source nodes from the pipeline supervisor when replication was stopped. + + + MB-18961 + Reduced default GoXDCR DCP checkpoint interval to 10 minutes for better + progress during disruptions. + + + MB-10093 + The XDCR network bandwidth can now be tuned using the Bandwidth Usage Limit + parameter either from the Web Console or by using the couchbase-cli xdcr-replicate + command. + + + MB-21879 + Replication was stuck when some DCP streams were inactive for a long period + and resulted in a race condition when the server tried to restart the inactive DCP + streams. + + + +
      +

      +

      New Supported Platforms

      +

      This release adds support for the following platforms:

        +
      • Oracle Linux 7
        • +
      • SUSE Linux Enterprise Server (SLES) 12
        • +
      • Ubuntu 16.04
        • +
      • Windows Server 2016
        • +
      See for the complete list of + supported platforms.

      + +

      Deprecated Features and Platforms

      +

      Starting this release, the following platforms are no longer supported:

        +
      • Amazon Linux 2014.03
        • +
      • macOS 10.10
        • +
      • Ubuntu 12.04
        • +
      • Windows Server 2008
        • +
      See for the complete list + of supported platforms.

      +

      The following functionality is deprecated, will be deprecated or is unsupported.

      + + + + + + + Function + Description + + + + + CLI setting-ldap command + This command is deprecated in this release. Use the admin-role-manage command + instead. + + + BASE64() + + The BASE64() function is deprecated since version 4.5; use + the function BASE64_ENCODE() instead.

      Also, use the + BASE64_DECODE() function to reverse the encoding done by + BASE64_ENCODE().

       +       + + Server-side moxi proxy is deprecated in Couchbase Server + and CLI + Use Couchbase client SDKs or client-side moxi in your applications. + + + CRAM-MD5 for bucket authentication + Use SCRAM protocol supporting clients for bucket authentication starting + Couchbase Server version 4.5. + + + Deprecate XDCR API from /internalSettings REST + endpoint in 4.5 + /Settings/replications REST endpoint will be available + for all replication settings and provides the same functionality. + + + N1QL: Deprecated the use of path-expressions in FROM and + JOIN clauses. + FROM and JOIN clauses should use only keyspaces. Paths can be used for + expressions in other clauses, operators, projections, and so on.

      For example: + SELECT count(*) FROM `travel-sample`.schedule; can be + rewritten as: + SELECT count(*) FROM `travel-sample` WHERE schedule IS NOT MISSING;

       +       + + CAPI based XDCR is deprecated. + CAPI based XDCR is deprecated and will be removed in a future version of + Couchbase Server.

      Note that the support for Elasticsearch Plugin has not been + removed. However, the Elasticsearch Plugin may be modified in the future to + use an alternative connection method such as DCP or XMEM XDCR (XDCR + v2).

       +       + + Couchbase Enterprise Backup and Restore tool + The tools, cbbackup and cbrestore, are + deprecated from the Enterprise Edition. We recommend replacing these old tools + with the new enterprise backup and restore tool introduced in version 4.5, cbbackupmgr. + + + Standard Global Secondary Indexes + A new high performance storage engine for GSI, Plasma, replaces the existing + GSI storage engine which is deprecated. + + + +
      +

      +
      + +       diff --git a/content/release-notes/relnotes-v5.1.x.dita b/content/release-notes/relnotes-v5.1.x.dita new file mode 100644 index 0000000000..b2133498c8 --- /dev/null +++ b/content/release-notes/relnotes-v5.1.x.dita @@ -0,0 +1,263 @@ + + + + Release Notes for Couchbase Server 5.1.x + +

      Couchbase Server 5.1.0 was released in February 2018. In addition to bug fixes, this release + includes enhancements to certificate-based authentication and the + cbbackupmgr utility. The following sections list the fixed issues, known + issues, and deprecated items in this release.

      +
      + Major Changes from Version 5.0 + +
      +
        +
      • Support for Debian 9 has been added.
        • +
      • Certificate-based authentication now supports multiple prefixes.
        • +
      +
      Known Issues + + + + + + + Issue + Description + + + + + MB-27688 + The query service does not support http/2 protocol. Consequently, if you want + to use CURL, you must pass "--http1.1" as one of the parameters + when using CURL to execute the query via the REST API. + + + +
      + +
      +
      Fixed Issues + +

      Cross Datacenter Replication (XDCR)

      + + + + + + + Issue + Description + + + + + MB-27850 + Whenever simultaneous replications were set up using XDCR, it was observed that data was not replicated via the second replication stream. This was basically due to rules enforced to manage the execution of simultaneous mutations to the document. + + + MB-25878 + When the source node was upgraded in a unidirectional replication from 4.1 to 5.x, the GoGC global setting was set to 0 instead of a higher number. + + + +
      + +

      Data Service

      + + + + + + + Issue + Description + + + + + MB-26126 + Statistics for non-resident documents could be incorrect when operating on + documents which contained system XATTRs.

      When using Couchbase Server 5.0 with Sync + Gateway 1.5+ and the Mobile Convergence feature was enabled in Sync Gateway + (enable_extended_attributes), the count of documents which are not resident could + be under-reported in the UI.

      +       + + MB-26037 + Rebalance failed initially if the destination node for a vBucket required a + rollback.

      During a rebalance, if the destination node for a particular vBucket + already had a previous, outdated version of that vBucket then the rebalance failed + initially.

       +       + + MB-25938 + If a document contained Extended Attributes and a Key/Value mutation request changed one of those XATTRs, the previous XATTR values were written to in a non-atomic way. This means that there was a potential race if another concurrent request was also accessing the same document at the same time - the other request could read a partial or corrupted value for the document. + + + MB-25935 + The data service incorrectly returned TEMP_FAIL for locked keys instead of ERROR_LOCKED. + + + MB-25785 + A backup resume operation on ephemeral buckets failed if there had been a data rollback during the previous backup that failed. + + + +
      + +

      Index Service

      + + + + + + + Issue + Description + + + + + MB-26085 + Rebalancing the index nodes failed with a timeout when the planner took a long time to execute. + + + +
      + +

      Query Service

      + + + + + + + Issue + Description + + + + + MB-27360 + On Windows platform, pasting a 1000+ character query into the cbq shell resulted in a partial query being pasted. This issue was observed after the query had been pasted a few times (>3). + + + MB-25901 + The ORDER BY operator sometimes continued to sort even after the operation was stopped leading to an exception. + + + +
      + +

      Search Service

      + + + + + + + Issue + Description + + + + + MB-25915 + Deleting FTS indexes during a rebalance operation may have produced a rare + situation that resulted in an unbreakable loop with log message: " + rebalance: waitAssignPIndexDone, awaiting a stats sample grab for + pindex " + + + +
      + +

      Tools

      + + + + + + + Issue + Description + + + + + MB-27459 + The cbbackup utility corrupted documents with extended attributes(XATTRs). + + + MB-27366 + The cbbackupmgr utility failed to backup directly from couchstore-files when the bucket was created on a version prior to 4.6 and threw the following error: + “error: could not read couchstore file due to unsupported file format version; exception: unpack requires a string argument of length 18” + + + MB-27270, MB-27279 + Couchbase Server version 5.0 introduced a new storage system “plasma” available only in the Enterprise Edition. When using the Enterprise Edition, you had a choice of “plasma” or “forestDB” for storage and the default was set to “plasma”. Since “plasma” storage was not available in the Community Edition, no storage was allocated when other services were started. This has been fixed and the default storage in the Community Edition now points to ForestDB. + + + MB-25785 + A backup resume on ephemeral buckets failed if there had been a data rollback during the prior backup that failed. + + + +
      + +

      Web Console

      + + + + + + + Issue + Description + + + + + MB-27733 + When creating more than 10 views, the confirmation dialog informing you of the performance impact was not clickable. This has been fixed and the dialog is now clickable. + + + MB-27221 + From the Query Monitoring tab, sorting the completed requests on the “run at” column did not appear to work as expected. This was because the “run at” column did not display the date, making the result of the sorting operation appear incorrect, since the same time on two different dates would appear in different spots on the list. This has been fixed by adding a clickable tooltip to the “run at” time which displays the complete time including the date. + + + MB-25959 + The Query Workbench incorrectly displayed a warning for a valid index creation statement. This was due to an incorrect handling of the 'bind' expression by the client-side parser. + + + +
      + + +
      + + + +       diff --git a/content/release-notes/relnotes.dita b/content/release-notes/relnotes.dita index fc1b91998d..759af91b6f 100644 --- a/content/release-notes/relnotes.dita +++ b/content/release-notes/relnotes.dita @@ -1,20 +1,23 @@ - Release Notes for Couchbase Server 5.0.x + Release Notes for Couchbase Server 5.5.x -

      Couchbase Server 5.0 delivers exciting capabilities for cross datacenter replication, - security, query, search, tools, and application development. Take a look at What's new for a list of new features and - improvements that are available in this release.

      -
      Release 5.0.1 -

      Couchbase Server 5.0.1, released in December 2017, supersedes version 5.0.0 released - earlier this year. Version 5.0.1 is the first maintenance release in the 5.0.x series for - Couchbase Server.

      -

      Fixed Issues

      +

      Couchbase Server 5.5.x is a significant update of the Couchbase Data Platform. This release + specifically highlights agility, performance, and manageability and includes enhancements to + existing services as well as new server and SDK components.

      +

      Take a look at for a list of new features and + improvements that are available in this release.

      + +
      Release 5.5.1 -

      Cross Datacenter Replication (XDCR)

      - +

      Couchbase Server 5.5.1, released in August 2018, is the first maintenance release in the 5.5.x series for + Couchbase Server.

      + +

      Fixed Issues

      + +

      Data Service

      +
      @@ -26,20 +29,56 @@ - MB-27162 - Revision-based conflict resolution did not work as expected as the revision - number was not updated for a create that immediately followed a delete operation. - The revision number of a deleted document was updated after the item was added to - the checkpoint queue, which meant that when you recreate that document, the revision - number was not being replicated intra-cluster or via XDCR. This has been fixed. + MB-30868 + Fixed an issue where the logs collected from the UI did not use the same salt across all nodes even though they were collected at the same time. + + + MB-30610 + Fixed an issue where the Memcached process was unable to start if IPv6 protocol stack was disabled. +
      -

      Security

      - +

      Index Service

      +
      + + + + + + Issue + Description + + + + + MB-30870 + During warmup, one of the indexes with low number of items triggered an issue + and caused the internal garbage collector to run forever. Thus causing the entire + index service to stall during warmup. This issue was observed when using Plasma + storage engine. + + + MB-30823 + Fixed an issue where the index service crashed with a "slice bounds out of range" error. + + + MB-30672 + Fixed an issue where upgrading from 4.x to 5.x version of Couchbase Server caused an outage for GSI standard indexes. This was observed during rolling upgrade from ForestDB to Plasma. When the last ForestDB index service node was taken out of the cluster, the cluster compatibility mode changed to Plasma and the index service restarted. + + + +
      + +

      Installation

      + @@ -51,86 +90,124 @@ - MB-27203 - The "default" bucket is a bucket that in - versions prior to 5.0 was either suggested or automatically created for users during - the install and configuration process with a goal of being compatible with memcached - out of the box. As its intention was to allow users to quickly get started with - Couchbase it permitted anonymous access, consistent with memcached behavior. - Starting 5.0, all buckets require authenticated access.

      However, there was a bug in 5.0.0 - where under certain circumstances anonymous access could be allowed to the bucket. - This issue only affected a bucket with the name "default" and - could only be triggered by an authenticated user with administrative permissions. This - issue has been fixed.

      + MB-30413 + Uninstalling Couchbase Server on a Debian-based systemd distribution such as Ubuntu 16 using "apt-get remove" and then running "systemctl status couchbase-server" reported that the service still exists but is "masked". This means the service was explicitly disabled in such a way that it could not be started, automatically or by hand. As a result, subsequent installations of any version of Couchbase Server failed as the service was rendered unstartable.
      -

      Tools

      - - - - - - - Issue - Description - - - - - MB-26556 - The couchbase-cli cluster-init command failed to - initialize all services in the Community Edition as the default storage mode - incorrectly pointed to Plasma, which is available in the Enterprise Edition, - instead of ForestDB. - - - -
      -

      + +

      Tools, Web Console (UI) and REST API

      + + + + + + + Issue + Description + + + + + MB-30636 + Fixed an issue where an unexpected token error was seen when accessing the Web Console in Couchbase Server version 5.5.0. + + + MB-30589 + Fixed an issue where the warning message on the Web Console showed an incorrect flag for the removal of dedicated bucket port as --remove-port instead of --remove-bucket-port. + + + +
      +
      -
      Release 5.0.0 -

      Couchbase Server 5.0.0 was released in October 2017. This release includes new features and behavior changes. The following sections list the behavior changes, fixed issues, known issues, and deprecated items.

      -

      Major Behavior Changes Compared to Version 4.6

      -

      Here are the behavior changes in the product, compared to the previous (4.6.x) release.

      -

      -

        -
      • Newly created buckets in 5.0 (including sample buckets) do not have any passwords - associated with them. To access a bucket, a user must be created with appropriate roles - to access using RBAC. See Bucket Full Access for details.
        • -
      • Important: Many 5.0 features require the latest versions of Couchbase SDK. - It is critical to update the SDK before updating to 5.0. Upgrade according to the SDK - versions listed in the SDK Compatibility chart below.
        • -
      • The TAP protocol which was previously deprecated, is now removed. TAP is an internal - protocol that streams information about data changes between cluster nodes. TAP is - replaced with a new protocol called Database Change Protocol (DCP).
        • -
      • Starting 5.0, port based buckets can only be created now through the bucket CLI - command. Use the command to create a port based bucket. Server-side moxi which - drives port based bucket creation has been deprecated. Consider using client-side moxi - instead.
        • -
      • For Windows, .exe installers which were previously deprecated are - now removed and replaced with industry-standard MSI installer files. Consequently, any - upgrade for the Windows platform from pre-5.0 version to 5.0 will need to be done in a - rolling upgrade fashion. See for details.
        • -
      • Re-designed Couchbase Web - Console.
        • -
      -

      + +
      Release 5.5.0 +

      Couchbase Server 5.5.0 was released in July 2018.

      + +

      Major Behavior Changes

      + + + + + + + + Issue + Description + + + + + MB-28778 + The default number of Data Service worker threads has been increased; from 3/4 + of the number of available CPU threads, to 7/8 of the number of available CPU + threads (minimum of 4 in both cases). This makes better use of the available CPU + resource, particularly on larger systems, as such users may see an increase in CPU + utilization on such systems. + + + MB-28417 + The permissions assigned to the cluster_admin role have been downgraded. As a + result, The cluster_admin role no longer has FTS write permissions used to create + FTS searches.

      Workaround: Add the fts_admin role as + required.

       +       + + MB-27173 + +

      Prior to 5.5, the mctimings command defined the request + start as the time when the 24 byte request header had been read by the Data + Service. However, many requests contain a body which also needs to be read + before a request can be processed. As such, the reported duration didn't + accurately reflect how long the server actually spent processing a request, if it + was still waiting for the body to be received from the network.

      +

      In 5.5, the definition of request start has been changed to be when the + header and body have both been read. This has the effect of more accurately + reflecting the time spent by the Data Service in servicing a request, as the time + the body spends in transit over the network is no longer included. As a result + command timings for affected request will have shorter durations compared to how + they were measured before 5.5.

      +       +       + + MB-27060 + The Couchbase Server 5.5 Windows installer package no longer bundles the DLL + dbghelp.dll which is required by Breakpad to produce dumps on + Windows platforms. Starting version 5.5, Breakpad will pick the DLL (dbghelp.dll) + that is shipped with the OS. + + + +
      + -

      Known Issues and Limitations

      -

      The following table lists some of the known issues in this release.

      - +

      New Supported Platforms

      +

      This release adds support for the Debian 9 platform. See Supported Platforms for the + complete list of supported platforms.

      + + +

      Deprecated Features and Platforms

      +

      The following functionality is deprecated, will be deprecated or is unsupported.

      +
        +
      • Server side Moxi or buckets with custom server side Moxi ports are deprecated and may be + removed from the product in the future.
        • +
      • The ability to create a bucket with a Moxi port has been deprecated. The CLI commands + have been updated to remove Moxi ports for bucket create and bucket edit operations.
        • +
      + +

      Known Issues

      + +

      Administration/Cluster Management

      +
      + + + + + + Issue + Description + + + MB-23074 Summary: Performance issues may be observed when running Couchbase - Server on CentOS 7.3 with kernel 3.10.0-514.6. + Server on CentOS 7.3 with kernel 3.10.0-514.6. + + + MB-17571 + Summary: On an undersized node, the default memory quota assigned to all + the selected services by the server might result in a failure.

      Workaround: + Adjust the memory allocations appropriately to work around this issue.

       +       + + + +
      +

      Data Service

      + + + + + + + Issue + Description + + + + + MB-30610 + Summary: Memcached process unable to start if IPv6 protocol stack + has been disabled. + Workaround: There are two generally accepted methods to disable IPv6 on recent Linux distributions: +
        +
      1. Disable the entire IPv6 protocol stack - typically by adding a boot-time grub parameter: ipv6.disable=1.
        2. +
      2. Disable assignment of IPv6 addresses to interfaces - either by setting the boot-time grub parameter: ipv6.disable_ipv6=1, + or dynamically by setting the appropriate net.ipv6.conf sysctl properties.
        3. +
      +

      Only the first method to disable IPv6 will cause Memcached to not start correctly. + As a workaround you can disable IPv6 using the second method instead.

      +       - MB-22679 - Summary: The Full Text Search service doesn't delete pindex files when - deleting the index after a MOSS crash. + MB-30074 + Summary: The data node is in a pending state after memcached + crashes. - MB-21772 - Summary: Full Text Search queries fail under high load on multi-node - cluster due to a large number of connections. + MB-29809 + Summary: The last_modified can be incorrect if a bucket + was upgraded and contains old documents. - MB-12000 - Summary: The rebalance progress reported during delta recovery is - confusing as certain phases like warmup are not tracked. + MB-29227 + Summary: Couchbase Server can get into a livelock state due to high + checkpoint memory usage.
      -

      This section lists some of the limitations in different areas across the product:

      - +

      Eventing Service

      +
      - + - Area + Issue Description + + + MB-29841 + Summary: The debugger is unable to apply the source map generated by the + transpiler. A recent update in the Chrome browser version 66.0.3359.181 is causing + this issue. Since Google doesn't allow previous versions of Chrome to be downloaded, + you can download previous versions of Chromium. The debugger works fine with source + maps on Chromium version 63.0.3239.0.

      Note: Our fix for Chromium is tagged to + 69.0.3466.0~157. As per the Chrome release calendar, 69.x is set to release on + September 4th 2018. The fix has been merged to Chrome Canary and is available at + https://www.google.com/chrome/browser/canary.html

             - Upgrade - -

      If you are using GSI indexes with a previously GA'd release of Couchbase (such as - 4.x) or Couchbase 5.0 Beta versions, we suggest that you drop the older index - definitions and recreate them using 5.0 GA version with Plasma, the newly - introduced storage engine for GSI.

      -       + MB-29360 + Summary: Multiple mutations are observed for a single document update + when using Sync Gateway, leading to the OnUpdate handler being triggered multiple + times. +       + + MB-29308 + Summary: The eventing service may mark redacted data + incorrectly. + + + MB-29271 + Summary: When a rebalance is in progress the Eventing service hangs when + memcached is killed on data and eventing nodes. + + + MB-28555 + Summary: The Eventing service currently does not provide the ability to + specify a port range. Instead, the OS arbitrarily allocates a port for the Chrome + Debugger. + + + MB-28414 + Summary: The eventing service misses some mutations with non-default + vBuckets (535,1001 etc). + + + MB-28120 + Summary: The eventing service rebalance progress jumps from 24% to + Finish. + + + MB-28010 + Summary: The execution_stats.on_update_failure are not + counted in the Failures stats displayed in the Web Console. + + + MB-27814 + Summary: When there are multiple functions being deployed, undeployment + does not happen until all functions are deployed. + + + MB-27559 + Summary: Benign panics can be seen in the eventing service logs during + undeployment. + + + + +
      +

      Full-text Search Service

      + + + + + + + Issue + Description + + + + + MB-29967 + Summary: In some circumstances, the Search engine uses more memory than + its defined quota. - N1QL Application Continuity:

      In 5.0 Beta 1 release, N1QL introduced multiple - performance enhancements enabled by a new internal protocol between Query and - Index services. While upgrading large Couchbase cluster deployments, the cluster - may be in a state where different Couchbase services are running on different - versions. In such scenarios, this feature ensures seamless continuity for N1QL - queries and applications irrespective of whether the query and indexing services - are running a 4.x or 5.0 version. The N1QL clients can avail the 5.0 features and - performance only when the issued query is processed by 5.0 query and indexing - services.

       -       - - Many 5.0 features require the latest versions of Couchbase SDK. It is critical - to update the SDK before updating to 5.0. Upgrade according to the SDK versions - listed in the SDK - Compatibility chart. - - - We’ve replaced the Windows .exe installers with - industry-standard MSI installer files in this release. Consequently, only rolling - upgrades from 4.x or earlier versions to 5.0 are supported. - - - SDK Compatibility - Many 5.0 features require you to upgrade your client SDK versions. The current - versions that support the latest features are shown below, and regardless of needing - new features, it is always advised to upgrade to the newest SDK version.

      -

      - - - - - - - SDK - Version - Reference - - - - - Java - 2.5.1 - Release notes - - - .NET - 2.5.0 - Release notes - - - Node.js - 2.4.0 - Release notes - - - Python - 2.2.6 - Release notes - - - PHP - 2.4.0 - Release notes - - - Go - 1.2.5 - Release notes - - - C - 2.8.0 - Release notes - - - -
      -

      + MB-28847 + Summary: The memory usage during index build overshoots the RAM + quota. + + + MB-27429 + Summary: Scorch indexes were found to contain duplicate pindexes. + + + MB-25714 + Summary: On Windows platform, a very high memory/CPU usage may be + observed when search service is indexing. -

      Fixed Issues

      -

      This section lists some of the important fixed issues in this release.

      -

      Data Service

      - - - - - - - Issue - Description - - - - - MB-23976 - The TOUCH command modifies a document and updates a CAS value too. However, - the result of TOUCH incorrectly used a pre-updated CAS value and did not reflect - the updated CAS. - - - MB-20091 - Improvements to ForestDB's thread handling for improved compaction. - - - MB-16150 - Fixed inconsistency in curr_items with full reviction on Data Greater than - Memory (DGM). - - - MB-13087 - When an item is locked and there’s actually a lot of load on the server, the - server now returns a PROTOCOL_BINARY_RESPONSE_LOCKED response to the client - application if the client application has enabled extended error codes. - - - -
      -

      Full Text Search Service

      +

      Indexing Service

      +
      @@ -481,44 +454,421 @@ - MB-23271 - To improve readability, the stat keys in section mossScope are now sorted. - + MB-30207 + Summary: An error is seen during a SUM aggregate pushdown when the entry + value is greater than MaxInt64. + + + MB-30011 + Summary: The rebalance progress in percentage during GSI swap rebalance + doesn't always increase linearly. + + + MB-19869 + Summary: Rebalance fails when taking out failed over nodes running + views, in certain circumstances. + + + +
      + +

      Install and Upgrade

      + + + + + + + Issue + Description + + + + + MB-30354 + Summary: Upgrading from 4.x to 5.x version of Couchbase Server causes an + outage for GSI standard indexes. This is observed during rolling upgrade from + ForestDB to Plasma. When the last ForestDB index service node is taken out of the + cluster, the cluster compatibility mode changes to Plasma and index service + restarts. + + + MB-30227 + Summary: In-place (offline) upgrades Server on Windows from 5.0.x or + 5.1.x to later releases will fail.

      Workaround: Manually uninstall 5.0.x / + 5.1.x from your Windows machines prior to installing 5.5.0 or later Server + releases.

       +       + + +
      + +

      Query Service

      + + + + + + + Issue + Description + + + + + MB-29391, MB-29393 + Summary: Large numeric values are reported incorrectly. For example, + the modulo operation or a scan for min int64 value returns an incorrect result in + case of int64. + + + +
      + +

      Security

      + + + + + + + Issue + Description + + + + + MB-26421 + Summary: The default "administrator" user doesn't show up in list of local + users displayed on the Web Console under the Security -> Users tab. + + + +
      + +

      Tools, Web Console (UI), and REST API

      + + + + + + + Issue + Description + + + + + MB-30589 + Summary: The UI displays an incorrect flag in the warning message about + the removal of dedicated port for a bucket. It displays + --remove-port instead of + --remove-bucket-port. + + + + +
      + +

      Analytics Service (Developer Preview)

      + + + + + + + Issue + Description + + + + + MB-30007 + Summary: It is possible for a CONNECT BUCKET statement to fail + temporarily with the error message "The vbucket belongs to another + server(0x7)" when rebalancing Data nodes.

      Workaround: + Retry the operation.

       +       + + MB-29542 + Summary: The Analytics stats for failed record count was + incorrect. + + + MB-28544 + Summary: The UNION ALL operation does not work for all query types. For + example, a query of the form shown below is not parsed correctly. + (SELECT ... FROM ... WHERE ...) +UNION ALL + (SELECT ... FROM ... WHERE ...)

      Workaround: + Remove the parentheses around the first subquery block. + SELECT ... FROM ... WHERE ... +UNION ALL + (SELECT ... + FROM ... + WHERE ...)

      +       + + +
      + + +

      Fixed Issues

      + + + +

      Data Service

      + + + + + + + Issue + Description + + + + + MB-29227 + Couchbase Server could get into a livelock state due to high checkpoint memory + usage. + + + MB-29205 + There were a number of issues in Couchbase Server which caused rebalance to + fail from time to time. These have been fixed. + + + MB-29040 + Rebalance failed with an "Invalid Format specified for DCP_DELETION" error when + data or eventing nodes were swapped in and out of a cluster. + + + MB-28868 + The index service went through a race condition in the storage engine which led + to a crash. The crash was most likely to happen if the index service was restarted + with existing build indexes and they continued to process mutations after the index + service restarted. + + + MB-28468 + The full-text search service repeatedly attempted to setup DCP streams to + non-existing vBuckets. + + + +
      + +

      Eventing Service

      + + + + + + + Issue + Description + + + + + MB-29235 + The Eventing service did not retry bucket operation failures that were + determined by the appropriate LCB macro to be retriable. + + + MB-29147 + Rebalance of KV/Eventing nodes can hang if the cluster contains Eventing nodes due to stale authentication info with the GoCB SDK. + + + MB-28968 + Rebalance of data or eventing nodes would hang if the cluster contained + Eventing nodes. + + + MB-28779 + If the handler code was not idempotent, the destination bucket contained more + docs than expected after a rebalance. + + + MB-28775 + On Windows, orphan processes with .rbf extension were + found to be running even after uninstalling Couchbase Server. + + + MB-28667 + Under some circumstances, like a function undergoing bootstrap or buckets + referenced by the function (source, metadata, destination) get flushed or deleted, + the function could not be undeployed. + + + MB-28550 + After restoring from an enterprise backup of eventing functions, eventing + service would not process mutations in the destination cluster. + + + MB-28520 + Deployment fails if the size of the handler code is greater than 1MB. + + + MB-28315 + Redeploying a function resulted in the application log being truncated instead + of appending to the existing log. + + + MB-27679 + The Eventing service can crash when processing documents in source buckets with + size is greater than 1 MB. + + + MB-27491 + Failed rebalance, when retried, could hang. + + + MB-27454 + Rebalance-in of a data(KV) node after recovery from failover + may hang if eventing service is processing mutations. + + + +
      + +

      Index Service

      + + + + + + + Issue + Description + + + + + MB-29405 + Rebalance would hang during index service rebalance. + + + MB-28139 + If the index files were corrupted, the index service would continue to crash + and required a failover or a manual deletion of the corrupted index files to + recover. + + + + +
      + +

      Query Service

      + + + + + + + Issue + Description + + + + + + MB-27815 + The array indexing plan incorrectly included filter covers. - MB-22560 - The FTS ports now adjust SSL cipher levels based on the - COUCHBASE_SSL_CIPHER_LIST environment variable. SSL ciphers can be set to LOW, - MEDIUM, or HIGH(default) levels. + MB-27360 + Fixed an issue on Windows platform where a query with more than 1000 characters + could be pasted into the cbq shell only the up to 3 times. Subsequent paste + operations resulted in a partial query being pasted. - MB-21855 - Improved error messages for the following scenarios:
        -
      • Creating an index with a name that already exists.
        • -
      • Creating an alias without target index.
        • -
      + MB-25901 + Fixed an issue where the ORDER BY operator proceeded to sort even after being + stopped and could lead to crashes.       + + +
      + + + +

      Tools, Web Console (UI), and REST API

      + @@ -530,387 +880,68 @@ - MB-21594 - Duplicate partitions removed during rebalance to prevent OOM (Out Of Memory - error), and subsequent recovery problems. + MB-29980 + The logic to confirm unsafe stop rebalance operation on the Web Console has + been resurrected. - MB-21156 - As part of a number of improvements to logging, fixed an issue where - couchdb.log wrapped too quickly. + MB-28337 + Cbbackup, cbtransfer and + cbrecovery can incorrectly timeout and in rare cases cause data + corruption. - MB-20297 - For an index that was built with defer_build:true, the CREATE INDEX statement - on the Indexes tab showed an invalid N1QL statement that could not be copied to run - as is using cbq or the Query Workbench. + MB-27719 + The cbrecovery command threw an exception when recovering + data from a cluster that had failed over nodes. + -

      Installer

      - - - - - - Issue - Description - - - - - MB-22283 - Installing and running Couchbase Server as a non-root and non-sudo user is - now supported. - - - -
      - Query Service

      - - - - - - - Issue - Description - - - - - MB-25103 - Query Service authentication failed when the password contained a colon (":") - character. - - - MB-24429 - When fetching data, if the data node failed to respond due to some reason, - the query could hang as there was no timeout specified. This has been addressed by - passing the query request timeout parameter when fetching data. - - - MB-23150 - The monitoring view for running queries provides details such as the node - that’s processing the request, the client IP that initiated the request, and the - running program type (i.e. WorkBench, CBC, Java, .NET, CBQ, etc.). This helps - isolate rogue queries and unauthorized access. - - - MB-22831 - Inserting a numerical value using a N1QL query caused the value to be stored - in scientific notation. - - - MB-22273 - In Couchbase clusters with multiple query service nodes, applications may get - incorrect results from prepared N1QL queries that use covering indexes. - - - MB-22119 - When using ARRAY expressions, query service may rarely panic with error - referring to ‘slice allocations’. - - - MB-22105 - The UPDATE, INSERT, UPSERT statements may fail with the error message: - "Panic: : runtime error: slice bounds out of range", when running with increased - pipeline_batch settings. - - - MB-22093 - The query engine may intermittently throw error code 12008 because of a - memory allocation failure. - - - MB-21928 - The command line shell for N1QL, cbq, now connects to 8091 with http and - 18091 with https by default when the port is not specified. - - - MB-19893 - The SELECT RAW did not take ORDER BY into account, thus the results were not - ordered. - - - MB-18769 - Fixed an issue where a covered query with meta().type in the WHERE clause - gave in correct results. - - - - -
      -

      Security

      - - - - - - - Issue - Description - - - - - MB-24088 - The secrets management feature fails to encrypt secrets larger than 4KB (such - as SSL certificates). This causes the cluster manager to unexpectedly terminate - when saving the cluster configuration to disk. - - - MB-15624 - You can now specify a SSL cipher list on Windows platform. - - - -
      -

      Tools

      - - - - - - Issue - Description - - - - - MB-25669 - On Windows platform, the cbimport tool failed to import a tab delimited CSV - file where the tab is specified using '\t' or '\\t'. - - - MB-20403 - The cbbackupmgr merge command did not function correctly and caused data - corruption under the following circumstances:
        -
      • The bucket being backed up was flushed between incremental backups.
        • -
      • The bucket being backed up was deleted and re-created in between incremental - backups.
        • -
      • The bucket had not been backed up between a document's deletion and the - metadata for that deleted document being purged.
        • -
      • There had been a failover between incremental backups of the bucket.
        • -
      -       - - MB-10093 - Xmem XDCR network bandwidth can now be throttled if desired. - - - -
      - Web Console

      - - - - - - - Issue - Description - - - - - MB-23251 - The AutoComplete feature for the password field on the Web Console has been - disabled. - - - MB-12480 - The text displayed when changing the RAM quota for a bucket was misleading - and has been fixed to reflect the actual ranges that the server will allow the - bucket to be sized within. - - - MB-9658 - The Web Console document browser for a bucket displays the ID and content - sample of the documents in the bucket. The UI displayed only the first 16 - characters and affected usability when viewing documents with keys longer than 16 - characters. This has been fixed to display the full key name and wrap to the next - line if needed. - - - -
      -

      XDCR

      - - - - - - - Issue - Description - - - - - MB-22514 - The XDCR target topology change detection process resulted in a large number - of open connections which caused a high CPU usage. This has been fixed by reducing - the number of connections to the target bucket. - - - MB-22172 - If a mutation needs to be resent, sent_time is updated so that next mutation - can be sent at the appropriate interval. - - - MB-21369 - Fixed a replication connection leak caused by a failure to close connections - to source nodes from the pipeline supervisor when replication was stopped. - - - MB-18961 - Reduced default GoXDCR DCP checkpoint interval to 10 minutes for better - progress during disruptions. - - - MB-10093 - The XDCR network bandwidth can now be tuned using the Bandwidth Usage Limit - parameter either from the Web Console or by using the couchbase-cli xdcr-replicate - command. - - - MB-21879 - Replication was stuck when some DCP streams were inactive for a long period - and resulted in a race condition when the server tried to restart the inactive DCP - streams. - - - -
      -

      -

      New Supported Platforms

      -

      This release adds support for the following platforms:

        -
      • Ubuntu 16.04
        • -
      • SUSE Linux 12
        • -
      • Oracle Linux 7
        • -
      • Microsoft Windows 2016
        • -
      See for the complete list of - supported platforms.

      + -

      Deprecated Features and Platforms

      -

      Starting this release, the following platforms are no longer supported:

        -
      • Microsoft Windows 7
        • -
      • Microsoft Windows 8
        • -
      • Microsoft Windows 2008
        • -
      • Amazon Linux 2014.03
        • -
      • Ubuntu Linux 12.04
        • -
      • Debian GNU/Linux 7
        • -
      • Mac OS X 10.8, 10.9, 10.10
        • -
      See for the complete list - of supported platforms.

      -

      The following functionality is deprecated, will be deprecated or is unsupported.

      - - - - - - - Function - Description - - - - - CLI setting-ldap command - This command is deprecated in this release. Use the admin-role-manage command - instead. - - - BASE64() - - The BASE64() function is deprecated since version 4.5; use - the function BASE64_ENCODE() instead.

      Also, use the - BASE64_DECODE() function to reverse the encoding done by - BASE64_ENCODE().

       -       - - Server-side moxi proxy is deprecated in Couchbase Server - and CLI - Use Couchbase client SDKs or client-side moxi in your applications. - - - CRAM-MD5 for bucket authentication - Use SCRAM protocol supporting clients for bucket authentication starting - Couchbase Server version 4.5. - - - Deprecate XDCR API from /internalSettings REST - endpoint in 4.5 - /Settings/replications REST endpoint will be available - for all replication settings and provides the same functionality. - - - N1QL: Deprecated the use of path-expressions in FROM and - JOIN clauses. - FROM and JOIN clauses should use only keyspaces. Paths can be used for - expressions in other clauses, operators, projections, and so on.

      For example: - SELECT count(*) FROM `travel-sample`.schedule; can be - rewritten as: - SELECT count(*) FROM `travel-sample` WHERE schedule IS NOT MISSING;

       -       - - CAPI based XDCR is deprecated. - CAPI based XDCR is deprecated and will be removed in a future version of - Couchbase Server.

      Note that the support for Elasticsearch Plugin has not been - removed. However, the Elasticsearch Plugin may be modified in the future to - use an alternative connection method such as DCP or XMEM XDCR (XDCR - v2).

       -       - - Couchbase Enterprise Backup and Restore tool - The tools, cbbackup and cbrestore, are - deprecated from the Enterprise Edition. We recommend replacing these old tools - with the new enterprise backup and restore tool introduced in version 4.5, cbbackupmgr. - - - Standard Global Secondary Indexes - A new high performance storage engine for GSI, Plasma, replaces the existing - GSI storage engine which is deprecated. - - - -
      -

      + +

      Analytics Service (Developer Preview)

      + + + + + + + Issue + Description + + + + + MB-27280 +

      It was possible for a rebalance of the Analytics nodes to + always fail if:

        +
      • a bucket was in the disconnected state, and
        • +
      • the Analytics nodes were at different DCP states.
        • +
      In that case a common DCP state (required for rebalancing) could not be achieved + as no new DCP mutations were received by the Analytics nodes.       +       + + +
      + +
      + + + +
      Release Notes for Older 5.x Versions +
        +
      • Release 5.1
        • +
      • Release 5.0
        • +
      +
      + + diff --git a/content/rest-api/compaction-rest-api.dita b/content/rest-api/compaction-rest-api.dita index a0ef25853e..7dec90d182 100644 --- a/content/rest-api/compaction-rest-api.dita +++ b/content/rest-api/compaction-rest-api.dita @@ -196,10 +196,11 @@
      Related Information
        -
      • To learn about the storage architecture and compaction, see .
        • +
      • To learn about the storage architecture and compaction, see + Storage.
      • To update the auto-compaction settings from the Web Console, see .
        • -
      • To update the bucket compaction settings using the CLI, see .
        • +
      • To update the bucket compaction settings using the CLI, see .
      diff --git a/content/rest-api/get-settings-indexes.dita b/content/rest-api/get-settings-indexes.dita index 9876adf3d3..78f4801b0e 100644 --- a/content/rest-api/get-settings-indexes.dita +++ b/content/rest-api/get-settings-indexes.dita @@ -1,7 +1,7 @@ - Retrieve the Global Secondary Index Settings + Retrieve GSI Settings To retrieve the global secondary index settings use GET /settings/indexes. @@ -40,13 +40,13 @@
      - Example Curl Command + Sample Curl Command

      The following example retrieves the global secondary index settings of the cluster that the node localhost is a part of.

      curl -X GET -u 'Administrator:password' 'http://localhost:8091/settings/indexes'

      - Example Response + Sample Response
      200
      @@ -70,4 +70,4 @@
      - \ No newline at end of file + diff --git a/content/rest-api/get-statement-indexes.dita b/content/rest-api/get-statement-indexes.dita new file mode 100644 index 0000000000..ff8f2b2448 --- /dev/null +++ b/content/rest-api/get-statement-indexes.dita @@ -0,0 +1,110 @@ + + + + + + Retrieve a GSI Statement + + + + To retrieve a GSI statement, use GET /getIndexStatement. + + + + +
      + + Description + + +

      + This endpoint returns a statement on Global Secondary Indexes. The specified IP address should be that of any cluster-node + that is running the Index Service; and the specified port must be 9102. +

      + +
      + +
      + + HTTP Method and URI + + + GET http://<host>:9102/getIndexStatement +
      +
      Response Codes + + + + + + + Response Code + Description + + + + + + 200 + Success. + + + 401 + Unauthorized. + + + +
      +
      +
      + + + Sample Curl Command + + +

      + The following example retrieves a GSI statement from node 10.142.180.101; which is running the Index Service, at port 9102. +

      + + curl -u Administrator:password -v -X GET http://10.142.180.101:9102/getIndexStatement + +
      + +
      + + + Sample Response + + +
      + +
      200 OK
      +
      +

      + The body contains the statement: this features a definition for each index. For example: +

      + + ["CREATE INDEX `def_airportname` ON `travel-sample`(`airportname`) WITH { \"defer_build\":true }", +"CREATE INDEX `def_city` ON `travel-sample`(`city`) WITH { \"defer_build\":true }","CREATE INDEX +`def_faa` ON `travel-sample`(`faa`) WITH { \"defer_build\":true }","CREATE INDEX `def_icao` ON +`travel-sample`(`icao`) WITH { \"defer_build\":true }","CREATE INDEX `def_name_type` ON +`travel-sample`(`name`) WHERE (`_type` = \"User\") WITH { \"defer_build\":true }","CREATE INDEX +`def_route_src_dst_day` ON `travel-sample`(`sourceairport`,`destinationairport`,(distinct (array +(`v`.`day`) for `v` in `schedule` end))) WHERE (`type` = \"route\") WITH { \"defer_build\":true }","CREATE INDEX +`def_schedule_utc` ON `travel-sample`(array (`s`.`utc`) for `s` in `schedule` end) WITH { \"defer_build\":true }", +"CREATE INDEX `def_sourceairport` ON `travel-sample`(`sourceairport`) WITH { \"defer_build\":true }", +"CREATE INDEX `def_type` ON `travel-sample`(`type`) WITH { \"defer_build\":true }","CREATE PRIMARY INDEX +`def_primary` ON `travel-sample` WITH { \"defer_build\":true }"] +
      +       + +
      401 Unauthorized
      +
      + The body of the response is empty. +
      +       +
      + +
      +       +       diff --git a/content/rest-api/get-status-indexes.dita b/content/rest-api/get-status-indexes.dita new file mode 100644 index 0000000000..c58bffc996 --- /dev/null +++ b/content/rest-api/get-status-indexes.dita @@ -0,0 +1,117 @@ + + + + + + Retrieve GSI Status + + + + To retrieve GSI status, use GET /getIndexStatus. + + + + +
      + + Description + + +

      + This endpoint returns status on Global Secondary Indexes. The specified IP address should be that of any cluster-node + that is running the Index Service; and the specified port must be 9102. +

      + +
      + +
      + + HTTP Method and URI + + + GET http://<host>:9102/getIndexStatus +
      +
      Response Codes + + + + + + + Response Code + Description + + + + + + 200 + Success. + + + 401 + Unauthorized. + + + +
      +
      +
      + + + Sample Curl Command + + +

      + The following example retrieves GSI status from node 10.142.180.101; which is running the Index Service, at port 9102. +

      + + curl -u Administrator:password -v -X GET http://10.142.180.101:9102/getIndexStatus + +
      + +
      + + + Sample Response + + +
      + +
      200 OK
      +
      + The response body contains status on each defined index. See below for a formatted example +
      +       + +
      401 Unauthorized
      +
      + The body of the response is empty. +
      +       +
      + +

      + Formatted status for one index contained in the successful response might appear as follows: +

      + + { + "defnId":4624455053989895893, + "instId":5414678899896559144, + "name":"def_airportname", + "bucket":"travel-sample", + "secExprs":["`airportname`"], + "indexType":"plasma", + "status":"Warmup", + "definition":"CREATE INDEX `def_airportname` ON `travel-sample`(`airportname`) WITH { \"defer_build\":true }", + "hosts":["10.142.180.101:8091"], + "completion":0, + "progress":0, + "scheduled":false, + "partitioned":false, + "numPartition":1, + "partitionMap":{"10.142.180.101:8091":[0]} +}, +
      +       +       diff --git a/content/rest-api/post-settings-indexes.dita b/content/rest-api/post-settings-indexes.dita index 1d70c1e778..072ec5e2f9 100644 --- a/content/rest-api/post-settings-indexes.dita +++ b/content/rest-api/post-settings-indexes.dita @@ -1,7 +1,7 @@ - Set the Global Secondary Index Settings + Set GSI Settings To set the global secondary index settings use POST /settings/indexes. @@ -87,12 +87,12 @@

      storageMode

       -

      The storage mode to be used for all global secondary indexes in the cluster. A value of forestdb sets the cluster-wide index storage mode to use ForestDB which can utilize both memory and persistent storage for index maintenance and index scans. A value of memory_optimized sets the cluster-wide index storage mode to use memory optimized global secondary indexes which can perform index maintenance and index scan faster at in-memory speeds. This setting can only be changed while there are no index nodes in the cluster. To change from standard GSI to memory optimized GSI or vice versa, you need to remove all the index service nodes in the cluster.

      +

      The storage mode to be used for all global secondary indexes in the cluster.* A value of plasma sets the cluster-wide index storage mode to use the Plasma storeage engine, which can utilize both memory and persistent storage for index maintenance and index scans. A value of memory_optimized sets the cluster-wide index storage mode to use memory optimized global secondary indexes which can perform index maintenance and index scan faster at in-memory speeds. This setting can only be changed while there are no index nodes in the cluster. To change from standard GSI to memory optimized GSI or vice versa, you need to remove all the index service nodes in the cluster.

      Possible values are:

        -
      • forestdb
        • +
      • plasma
      • memory_optimized

       @@ -100,6 +100,7 @@ +

      * Plasma and memory_optimized are options in the Enterprise Edition of Couchbase Server 5.5. If you are using the Community Edition, the default (and only) value is forestdb.

      Response Codes @@ -127,13 +128,13 @@
      - Example Curl Command + Sample Curl Command

      The following example sets the global secondary index settings.

      curl -X POST -u 'Administrator:password' 'http://localhost:8091/settings/indexes' -d 'indexerThreads=0' -d 'logLevel=info' -d 'maxRollbackPoints=5' -d 'memorySnapshotInterval=200' -d 'stableSnapshotInterval=5000' -d 'storageMode=forestdb'

      - Example Response + Sample Response
      200
      @@ -157,4 +158,4 @@
      -       \ No newline at end of file + diff --git a/content/rest-api/rbac.dita b/content/rest-api/rbac.dita index 2c0474b0bd..25b61ef826 100644 --- a/content/rest-api/rbac.dita +++ b/content/rest-api/rbac.dita @@ -1,49 +1,81 @@ - Role Based Admin Access (RBAC) - Full Couchbase administrators can manage role-based access for administrators (RBAC) - using REST API. + + + Role Based Admin Access (RBAC) + + + + Full Administrators can manage the Couchbase Role-Based Access Control (RBAC) system, + using the REST API. + + + This fine-grained access applies to the Enterprise Edition of Couchbase Server. The + Community Edition is limited to the roles of bucket_full_access, admin, and ro_admin; + see the Authorization page for further details. +
      - GET /settings/rbac/roles + + + GET /settings/rbac/roles + +

      Description

      -

      This command retrieves information about the available admin roles.

      +

      This command retrieves information about the available roles.

      Syntax:

      - $ curl -X GET http://user:password@127.0.0.1:8091/settings/rbac/roles + $ curl -X GET http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/roles
      +
      GET /settings/rbac/users

      Description

      -

      This command retrieves information about the current users and roles assigned to them.

      +

      This command retrieves information about the current users and the roles assigned to them.

      Syntax:

      - $ curl -X GET http://user:password@127.0.0.1:8091/settings/rbac/users + $ curl -X GET http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/users
      +
      PUT /settings/rbac/users/<user_id>

      Description

      -

      This command sets names and roles for the specified user IDs.

      +

      This command sets the name and roles for the specified user ID. Note that the new user's password is + expected to be accessible, when authentication is subsequently attempted, from an external domain.

      +

      Syntax: +

      + $ curl -X PUT http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/users/<user_id> +
      + +
      + PUT /settings/rbac/users/local/<user_id> +

      Description

      +

      This command sets name, role, and password for the specified user ID. Note that by these means, the new user is + defined locally to Couchbase Server (the user's password having been provided as a parameter to this command).

      Syntax:

      - $ curl -X PUT http://user:password@127.0.0.1:8091/settings/rbac/users/<user_id> + $ curl -X PUT http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/users/local/<user_id>
      + +
      DELETE /settings/rbac/users/<user_id>

      Description

      -

      This command deletes the user specified with an ID.

      +

      This command deletes the specified user.

      Syntax:

      - $ curl -X DELETE http://user:password@127.0.0.1:8091/settings/rbac/users/<user_id> + $ curl -X DELETE http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/users/<user_id>
      +
      POST /pools/default/checkPermissions

      Description

      -

      This command checks the permissions.

      +

      This command checks the permissions on specified Couchbase Server-resources that have been assigned to + the administrator whose credentials are specified in the command.

      Syntax:

      - $ curl -X POST http://user:password@127.0.0.1:8091/pools/default/checkPermissions + $ curl -X POST http://Administrator:adminpwd@127.0.0.1:8091/pools/default/checkPermissions
      @@ -51,89 +83,155 @@
      -
      Get list of available roles
      +
      Get a list of available roles
      GET /settings/rbac/roles
      -
      Returns array of objects like this one: - {"role":"admin", "name":"Admin", "desc": - "Can manage ALL cluster features including security."}

      or - with extra property "bucket_name":"default" if the role is - parametrized.

      +
      Returns an array of objects whose fields explain the available roles on the system. The + first and second objects in the array might appear as follows: + [ + { + "role": "replication_target", + "bucket_name": "*", + "name": "Replication Target", + "desc": "XDC replication target for bucket" + }, + { + "role": "query_external_access", + "name": "Query External Access", + "desc": "Can execute CURL statement" + }, + . + . +
      -
      Get a list of users and roles assigned to them.
      +
      Get a list of all current users and the roles they are assigned.
      GET /settings/rbac/users -

      The list does not include built-in users.

      -

      In the following example, the output of the command shows that John Doe was assigned to - the role of the Bucket Administrator for the default bucket, while Alice Smith is the - Replication Administrator.

      + +

      For example:

      - $ curl -X GET http://Administrator:asdasd@127.0.0.1:8091/settings/rbac/users + $ curl -X GET http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/users - [{"name":"John Doe","id":"johndoe","roles":[ - {"role":"admin"} - , - {"role":"bucket_admin","bucket_name":"default"} - ]},{"name":"alice-smith","id":"alicesmith","roles":[ - {"role":"replication_admin"} - ]}] +[ + { + "name": "John Smith", + "id": "johnsmith", + "domain": "local", + "roles": [ + { + "role": "cluster_admin" + } + ] + } +]
      +
      -
      Set names and roles for specified user IDs <user_id>
      +
      Set the name, user ID, and roles for a new user.
      PUT /settings/rbac/users/<user_id>
      -
      In this example, John Doe is assigned to be the Cluster Administrator and a Bucket - Administrator for the default bucket, while Alice Smith is assigned to be the - Replication Administrator: - $ curl -X PUT --data "name=John Doe&roles=cluster_admin,bucket_admin[default]" \ - http://Administrator:asdasd@127.0.0.1:8091/settings/rbac/users/johndoe - - [{"name":"John Doe","id":"johndoe,"roles":[ - {"role":"cluster_admin"} - , - {"role":"bucket_admin","bucket_name":"default"} - ]},{"name":"Alice Smith","id":"alicesmith","roles":[ - {"role":"replication_admin"} - ]}]
      +
      In this example, John Doe is assigned to be Cluster Administrator, and also to be Bucket + Administrator for the travel-sample bucket, with user ID of + johndoe: + $ curl -X PUT --data "name=John Doe&roles=cluster_admin,bucket_admin[travel-sample]" \ + http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/users/johndoe + +

      + Note that by means of the command above, the user is listed + by Couchbase Server as being in the External + authentication domain: this means that the user ID + specified here is expected to match a user ID by which they + are registered (with an appropriate password) on a remote (for example, an LDAP) system. If + the user ID specified in this command + does not so match, the user is permitted no access to Couchbase Server-resources. See below for an example of + adding user and roles by specifying a password to be stored locally. See + Authentication for more + information on defining users remotely. +

      +
      + +
      + +
      Set the name, user ID, roles, and password for a new user.
      +
      + PUT /settings/rbac/users/local/<user_id> +
      +
      In this example, John Smith is assigned to be the Cluster Administrator, + with password specified as jspassword, and user ID as + johnsmith: + + $ curl -X PUT --data "name=John Smith&roles=cluster_admin&password=jspassword" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/users/local/johnsmith + +

      + Note that by means of the command above, the user is assigned to the + Local authentication domain: this means that they have been registered on Couchbase + Server itself, with the password specified. +

      +
      +       +
      +
      Delete users
      DELETE /settings/rbac/users/<user_id>
      -
      In this example, Alice Smith was deleted as the Replication Administrator and the other - user, John Doe, remains as a Cluster administrator. - $ curl -X DELETE http://Administrator:asdasd@127.0.0.1:8091/settings/rbac/users/alicesmith - [{"name":"John Doe","id":"johndoe,"roles":[ - {"role":"cluster_admin"} - ]}]
      +
      In this example, the user identified by the user ID alicesmith is deleted. + $ curl -X DELETE http://Administrator:adminpwd@127.0.0.1:8091/settings/rbac/users/alicesmith
      -
      Check multiple permissions
      +
      Check permissions
      POST /pools/default/checkPermissions
      -
      Example: - $ curl -X POST --data 'cluster.bucket[default].stats!read,cluster.bucket[default]!write' \ - http://roadmin:asdasd@127.0.0.1:8091/pools/default/checkPermissions - {"cluster.bucket[default].stats!read":true,"cluster.bucket[default]!write":false}
      +
      The following example checks the authenticating administrator's + permissions on the travel-sample + bucket, for reading bucket-statistics and for writing to the bucket: + + $ curl -X POST --data 'cluster.bucket[travel-sample].stats!read,cluster.bucket[travel-sample]!write' \ + http://Administrator:adminpwd@127.0.0.1:8091/pools/default/checkPermissions + + +

      + Output might appear as follows: +

      + + { +"cluster.bucket[travel-sample].stats!read": true, +"cluster.bucket[default]!write": true +} + +
      -
      + + + -
      Reading the Log Output -

      Here are some output logs with comments to help you understand how to read such logs.

      +
      + + + Reading Log Output + + +

      + The following examples of log output are provided, with comments to aid understanding.. +

      + {"name":"John Doe","id":"johndoe","roles":[{"role":"admin"}]}] {'status': '200', 'content-length': '64', 'server': 'Couchbase Server', 'pragma': 'no-cache', \ 'cache-control': 'no-cache', 'date': 'Mon, 13 Jun 2016 10:35:28 GMT', 'content-type': 'application/json’} @@ -198,6 +296,6 @@
      - + diff --git a/content/rest-api/rest-api.ditamap b/content/rest-api/rest-api.ditamap index c590f8b3b6..e561b22f72 100644 --- a/content/rest-api/rest-api.ditamap +++ b/content/rest-api/rest-api.ditamap @@ -56,10 +56,24 @@ + + + + + + + + + + + + + + @@ -67,11 +81,6 @@ - - - - - @@ -83,12 +92,6 @@ - - - - - - @@ -106,12 +109,12 @@ + - diff --git a/content/rest-api/rest-auditing.dita b/content/rest-api/rest-auditing.dita index 4b669d0378..99992d462a 100644 --- a/content/rest-api/rest-auditing.dita +++ b/content/rest-api/rest-auditing.dita @@ -1,46 +1,99 @@ - Auditing APICouchbase Server APIs for audit management. + + + Auditing API + + + Couchbase Server APIs for audit management. + + - GET settings/audit -

      Returns information about the audit settings.

      - POST settings/audit/<settings> + GET settings/audit +

      + Returns information about the audit settings. +

      + POST settings/audit
      - GET /settings/audit -

      Description

      -

      This command retrieves information about the audit settings.

      -

      Syntax: + + + GET /settings/audit + + +

      + Description

      - $ curl GET http://user:password@127.0.0.1:8091/settings/audit + +

      + This command retrieves information about the audit settings. +

      + +

      + Syntax: +

      + + curl -X GET -u Administrator:password http://127.0.0.1:8091/settings/audit +
      - +
      - POST /settings/audit/<settings> -

      Description

      -

      This command configures the audit settings.

      -

      Syntax: -

      -

      $ curl POST http://user:password@127.0.0.1:8091/settings/audit/auditEnabled - Enable audit. Use true/false to set.

      -

      $ curl POST http://user:password@127.0.0.1:8091/settings/audit/logPathSet - the location for audit logs. The default location is - /opt/couchbase/var/lib/couchbase/logs. You can set the location with - REST, or with the Couchbase Web - Console.

      -

      $ curl POST http://user:password@127.0.0.1:8091/settings/audit/rotateInterval - Set the time in seconds for the log rotating interval.

      + + + POST /settings/audit + + +

      + Description +

      + +

      + This command configures the audit settings. +

      + +

      + Syntax: +

      + +

      + curl -X POST -d auditdEnabled=true -u Administrator:password http://127.0.0.1:8091/settings/audit + + Enable audit. The value of auditdEnabled can be either true or false. + +

      + +

      + curl -X POST -d logPath='/opt/couchbase/var/lib/couchbase/logs' \ +-u Administrator:password http://127.0.0.1:8091/settings/audit + + Set the location for audit logs. The default location on Linux is /opt/couchbase/var/lib/couchbase/logs. +

      + + +

      + curl -X POST -d rotateInterval=582300 -u Administrator:password http://127.0.0.1:8091/settings/audit + + Specify the time to elapse between log-rotations. The value of rotateInterval must be a number of + whole minutes, expressed as seconds: therefore, the specified number must be a multiple of 60. The acceptable range + is 900 (15 minutes) to 604800 (7 days), inclusive. +

      + + - - +       + + + + + diff --git a/content/rest-api/rest-bucket-create.dita b/content/rest-api/rest-bucket-create.dita index 66a7a395d6..59b42d6d4b 100644 --- a/content/rest-api/rest-bucket-create.dita +++ b/content/rest-api/rest-bucket-create.dita @@ -30,10 +30,10 @@
      HTTP method and URI - POST /pools/default/buckets + POST /pools/default/buckets
        -
      • Request data - List of payload parameters for the new bucket
        • +
      • Request data - List of payload parameters for the new bucket (in the body of the POST as x-www-form-urlencoded)
      • Response data - JSON of the bucket confirmation or an error condition
      • Authentication required - Yes
      @@ -96,7 +96,10 @@ name - Required parameter. Name for new bucket. This cannot be changed for a pre-existing bucket. + Required parameter. Name for new bucket. This cannot be changed for a pre-existing bucket. + A bucket name can only contain characters in the ranges of A-Z, a-z, and 0-9; with the + addition of the underscore, period, dash and percent characters; and can be no more + than 100 characters in length. parallelDBAndViewCompaction @@ -141,16 +144,48 @@ Optional Parameter. Integer from 2 to 8. Change the number of concurrent readers and writers for the data bucket. If you change this setting for a pre-existing bucket then it will be restarted, resulting in temporary inaccessibility of data while the bucket warms up. - - + compressionMode + Optional Parameter. + Specifies the compression-mode of the bucket. There are three options; off, passive and active. All three modes + are backwards compatible with older SDKs, where Couchbase Server will automatically uncompress documents for + clients that do not understand the compression being used. This option is only available for Couchbase and + Ephemeral buckets in Couchbase Enterprise Edition. + +

      +

      + +

      + Off: Couchbase Server will only compress document values when persisting to + disk. This is suitable for use cases where compression could have a + negative impact on performance. Please note it is expected that compression + in most use cases will improve performance. +

      + +

      + Passive: Documents which were compressed by a client, or read compressed + from disk will be stored compress in-memory. Couchbase Server will make + no additional attempt to compress documents that are not already compressed. +

      + +

      + Active: Couchbase Server will actively and aggressively attempt to compress + documents, even if they have not been received in a compressed format. This + provides the benefits of compression even when the SDK clients are not + complicit. +

      +       + + @@ -177,47 +212,75 @@
      Syntax

      Curl request syntax:

      - curl -X POST -u [admin]:[password] + curl -X POST -u [admin]:[password] http://[ip_address]:8091/pools/default/buckets -d name=[new-bucket-name] -d ramQuotaMB=[value] -d authType=[none | sasl] -d replicaNumber=[value] - -d proxyPort=[proxy-port] - http://[localhost]:8091/pools/default/buckets + -d proxyPort=[proxy-port]
      Example

      Curl request example:

      - curl -X POST -u Administrator:password \ - -d 'name=newbucket' -d 'ramQuotaMB=20'0 -d 'authType=none' \ - -d 'replicaNumber=2' \ - -d 'proxyPort=11215' \ - http://10.5.2.54:8091/pools/default/buckets -

      Curl request example to set conflict resolution type:

      -

      Set the parameter conflictResolutionType to lww during + curl -X POST -u Administrator:password http://127.0.0.1:8091/pools/default/buckets \ +-d name=newBucket -d ramQuotaMB=100 -d authType=none \ +-d replicaNumber=2 \ +-d proxyPort=11215 +

      Curl request example to set conflict resolution type. Set the parameter + conflictResolutionType to lww during bucket creation. For example, use the following command to create a bucket on the - source cluster:

      - curl -X POST -u Administrator:asdasd http://<ip_address>:8091/pools/default/buckets + source cluster: +

      + + curl -X POST -u Administrator:asdasd http://<ip_address>:8091/pools/default/buckets -d name=newBucketSource -d conflictResolutionType=lww -d authType=sasl -d ramQuotaMB=4096 -d saslPassword=passw0rd -d bucketType=couchbase +

      Use the following command to create a bucket on the destination cluster: - curl -X POST -u Administrator:asdasd http://<ip_address>:8091/pools/default/buckets + curl -X POST -u Administrator:asdasd http://<ip_address>:8091/pools/default/buckets -d name=newBucketDestination -d conflictResolutionType=lww -d authType=sasl -d ramQuotaMB=4096 -d saslPassword=passw0rd -d bucketType=couchbase

      + +

      + Curl request to set maximum Bucket Time-To-Live, and to establish a compression mode: +

      + + curl -X POST -u Administrator:password http://127.0.0.1:8091/pools/default/buckets \ +-d name=myTestBucket -d ramQuotaMB=100 \ +-d bucketType=couchbase -d maxTTL=20000 \ +-d compressionMode=Passive + +

      + Curl request to edit an existing bucket: +

      + + curl -X POST -u Administrator:password http://10.142.180.101:8091/pools/default/buckets/testBucket \ +-d bucketType=couchbase \ +-d autoCompactionDefined=false \ +-d evictionPolicy=valueOnly \ +-d threadsNumber=3 \ +-d replicaNumber=1 \ +-d compressionMode=passive \ +-d maxTTL=20000 \ +-d replicaIndex=0 \ +-d ramQuotaMB=100 \ +-d flushEnabled=0 +

      Raw HTTP request example:

      The parameters for configuring the bucket are provided as payload data. Each parameter and value are provided as a key/value pair where each pair is separated by an ampersand. Include the parameters setting in the payload of the HTTP POST request.

      - POST /pools/default/buckets + POST /pools/default/buckets HTTP/1.1 Host: 10.5.2.54:8091 Content-Type: application/x-www-form-urlencoded; charset=UTF-8 Authorization: Basic YWRtaW46YWRtaW4= Content-Length: xx name=newbucket&ramQuotaMB=20&authType=none&replicaNumber=2&proxyPort=11215 +
      Response diff --git a/content/rest-api/rest-bucket-stats.dita b/content/rest-api/rest-bucket-stats.dita index 3440770e1a..9c4a9e067a 100644 --- a/content/rest-api/rest-bucket-stats.dita +++ b/content/rest-api/rest-bucket-stats.dita @@ -147,13 +147,13 @@ Content-Length: nnn
      Example: zoom parameter

      The following example retrieves sample statistics from a bucket for the last minute.

      - curl -X GET -u admin:password -d zoom=minute http://localhost:8091/pools/default/buckets/bucket-name/stats + curl -X GET -u Administrator:password http://localhost:8091/pools/default/buckets/bucket-name/stats?zoom=minute

      The following example retrieves sample statistics from a bucket for the past day.

      - curl -X GET -u admin:password -d zoom=day http://localhost:8091/pools/default/buckets/bucket-name/stats + curl -X GET -u Administrator:password http://localhost:8091/pools/default/buckets/bucket-name/stats?zoom=day

      The following example retrieves sample statistics from a bucket for the last month.

      - curl -X GET -u admin:password -d zoom=month http://localhost:8091/pools/default/buckets/bucket-name/stats + curl -X GET -u Administrator:password http://localhost:8091/pools/default/buckets/bucket-name/stats?zoom=month diff --git a/content/rest-api/rest-cluster-autofailover-enable.dita b/content/rest-api/rest-cluster-autofailover-enable.dita index 41a7604c90..f5e1bf60ce 100644 --- a/content/rest-api/rest-cluster-autofailover-enable.dita +++ b/content/rest-api/rest-cluster-autofailover-enable.dita @@ -1,38 +1,101 @@ - Enabling and Disabling Auto-Failover - Auto-failover is enabled and disabled the POST /settings/autoFailover HTTP method and URI. + + + Enabling and Disabling Auto-Failover + + + + Auto-failover is enabled and disabled by means of the POST /settings/autoFailover HTTP method and URI. + + -
      HTTP method and URI -

      This is a global setting that applies to all clusters. Authentication is required to change this value.

      + +
      + + + HTTP method and URI + + +

      + The auto-failover setting is global, and therefore applies to all nodes in the cluster. Setting-changes require + authentication. +

      + POST /settings/autoFailover -

      Parameters include:

      +

      + The following parameters are used: +

      +
        -
      • enabled: Value is true or false. Indicates whether Couchbase Server - performs auto-failover for the cluster or not.
        • -
      • timeout: Positive integer between 30 and 3600. The number of seconds a - node must be down before Couchbase Server performs auto-failover on the node. Required if - enabled=true; optional when enabled=false.
        • +
      • + enabled=[true|false]. Enables or disables automatic failover. Default setting is false. +
        • + +
      • + timeout=number-of-seconds. Integer between 5 and 3600. Specifies the number of seconds + that must elapse, with a node unavailable, before automatic failover is triggered. Default setting is 120. +
        • + +
      • + maxCount=number-of-events. Specifies the maximum number of sequential auto-failover events to be + handled before administrator-intervention is required. An event might consist of one node failing over; or + alternatively, of an entire server group failing over. + The maximum allowed value is 3. The default value is 1. This parameter is only supported by Couchbase + Server Enterprise Edition. +
        • + +
      • + failoverServerGroup=[true|false]. + Enables or disables automatic failover for server groups. + Do not enable failover for server groups unless you have set up three or more server groups, and have the capacity + to absorb the load of any failed-over group. This parameter is ignored if the value of the enabled parameter + for automatic failover + is false. Default setting is false. This parameter is only supported by Couchbase + Server Enterprise Edition. +
        • + +
      • + failoverOnDataDiskIssues[enabled]=[true|false]&failoverOnDataDiskIssues[timePeriod]=number-of-seconds. + When enabled is true, timePeriod is an + integer between 5 and 3600: such settings allow automatic failover to be triggered when disk read-write attempts have failed continuously + throughout at least 60% of the specied time-period. The default for enabled + is false. If enabled is true, the default timePeriod is + 120 seconds. If enabled is false and the timePeriod is explicitly specified, + an error is returned. This parameter and its values are ignored if the value of the enabled parameter for + automatic failover is false. This parameter is only supported by Couchbase Server Enterprise Edition. +
        • +
      +
      Syntax

      Curl request syntax:

      - curl -u [admin]:[password] http://[localhost]:8091/settings/autoFailover -d [parameter] + curl -u [admin]:[password] http://[localhost]:8091/settings/autoFailover -d [parameter]
      -
      Example -

      Curl request example:

      - curl -i -u Administrator:password \ - http://10.5.2.54:8091/settings/autoFailover \ - -d 'enabled=true&timeout=600' +
      + + + Example + + +

      + Curl request example: +

      + + curl -i -X POST -u Administrator:password http://10.142.180.103:8091/settings/autoFailover \ + -d 'enabled=true&timeout=72' \ + -d 'failoverServerGroup=true&maxCount=2' \ + -d 'failoverOnDataDiskIssues[enabled]=true&failoverOnDataDiskIssues[timePeriod]=89'

      Raw HTTP request example:

      - POST /settings/autoFailover HTTP/1.1 + POST /settings/autoFailover HTTP/1.1 Host: 10.5.2.54:8091 Content-Type: application/x-www-form-urlencoded Authorization: Basic YWRtaW46YWRtaW4= @@ -43,15 +106,16 @@ enabled=true&timeout=600
      Response codes - HTTP/1.1 200 OK + HTTP/1.1 200 OK Server: Couchbase Server Pragma: no-cache Date: Sat, 18 Oct 2014 00:26:28 GMT Content-Length: 0 Cache-Control: no-cache -

      The possible errors include:

      - 400 Bad Request, The value of "enabled" must be true or false. -400 Bad Request, The value of "timeout" must be a positive integer bigger or equal to 30. +

      Possible errors include:

      + 400 Bad Request, The value of "enabled" must be true or false +400 Bad Request, The value of "timeout" must be a positive integer in a range from 5 to 3600 +400 Bad Request, The value of "failoverOnDataDiskIssues[timePeriod]" must be a positive integer in a range from 5 to 3600 401 Unauthorized This endpoint isn't available yet. diff --git a/content/rest-api/rest-cluster-autofailover-intro.dita b/content/rest-api/rest-cluster-autofailover-intro.dita index c6e2c9a692..051a8c137f 100644 --- a/content/rest-api/rest-cluster-autofailover-intro.dita +++ b/content/rest-api/rest-cluster-autofailover-intro.dita @@ -1,20 +1,36 @@ - Managing Auto-Failover - Auto-failover is managed with with the GET /settings/autoFailover HTTP method and URI. + + + Managing Auto-Failover + + + + Auto-failover is managed by means of the the GET /settings/autoFailover HTTP method and URI. + + -
      Description -

      This section provides information about retrieving, enabling, disabling and resetting - auto-failover.

      +
      + + Description + + +

      This section provides information about retrieving, enabling, disabling, and resetting + auto-failover. +

      - Auto-failover endpoints + + Auto-Failover Endpoints + + - - - + + + + HTTP method @@ -22,30 +38,92 @@ Description + GET + /settings/autoFailover - Retrieves automatic failover settings. Parameters include:
        -
      • Enabled=[true|false] : True to enable failover; false to - disable failover.
        • -
      • timeout=[value] : Integer between 30 and 3600. Specifies the - amount of time (in seconds) that a node is down before failover in - initiated.
        • -
      • count=[0|1]: Value is 0 or 1. It shows whether any node in a - cluster can be automatically failed-over. After one auto-failover occurs, the - count is set to 1 and Couchbase Server does not perform another auto-failover - for the cluster unless the count is reset to 0. To failover more than one node - at a time in a cluster, perform a manual failover.
        • -
      + + + Retrieves automatic-failover settings. Uses the following parameters: +

      +

      + +
        + +
      • + enabled. Indicates whether automatic failover is enabled (a value of true) or + disabled (a value of false). +
        • + +
      • + timeout. Returns an integer between 5 and 3600, which specifies the number of seconds set + to elapse, + after a node has become unavailable, before automatic failover is triggered. +
        • + +
      • + count. This parameter represents how many sequential auto-failover events have + occurred since the parameter + was itself last reset, to a value of 0, through administrator intervention. +
        • + +
      + +       +       + POST + /settings/autoFailover - Enables and disables automatic failover. To enable or disable failover, use the - enabled=[true|false] parameter. To specify the number of seconds - that a node must be down before initiating failover, use the - timeout parameter. + + + Enables and disables automatic failover. Uses the following paramenters: + +

      + +

      + +
        +
      • + enabled=[true|false]. Enables or disables automatic failover. +
        • + +
      • + timeout=number-of-seconds. Integer between 5 and 3600. Specifies the number of seconds + that must elapse after a node has become unavailable, before automatic failover is triggered. +
        • + +
      • + maxCount=number-of-events. Specifies the number of auto-failover events that are + sequentially handled before user intervention is required. An event might consist of one node failing over; or + alternatively, of an entire server group failing over. + The maximum allowed value is three. The default value is one. +
        • + +
      • + failoverServerGroup=[true|false]&maxCount=number-of-seconds. + Enables or disables automatic failover for server groups, and specifies the number of seconds that must elapse + after a node has become unavailable, before automatic failover is triggered. + Do not enable failover for server groups unless you have set up three of more server groups, and have the capacity + to absorb the load of any failed-over group. +
        • + +
      • + failoverOnDataDiskIssues[enabled]=[true|false]&failoverOnDataDiskIssues[timePeriod]=number-of-seconds. + Integer between 5 and 3600. Enables or disables automatic failover for any sustained disk read/write failure, + after the specied time-period has elapsed +
        • + + + +
      + +       +       POST diff --git a/content/rest-api/rest-cluster-autofailover-reset.dita b/content/rest-api/rest-cluster-autofailover-reset.dita index 4de5a27b47..a9d02292eb 100644 --- a/content/rest-api/rest-cluster-autofailover-reset.dita +++ b/content/rest-api/rest-cluster-autofailover-reset.dita @@ -1,41 +1,77 @@ - Resetting Auto-Failover - Auto-failover is reset with the POST /settings/autoFailover/resetCount HTTP method and URI. + + + Resetting Auto-Failover + + + + Auto-failover is reset by means of the POST /settings/autoFailover/resetCount HTTP method and URI. + + -
      HTTP method and URI -

      Resets the number of nodes that Couchbase Server has automatically failed over. A request - can be sent to reset the auto-failover number to 0. This is a global setting for all - clusters. Authenticated is required to change this value. No parameters are required.

      + +
      + + + HTTP method and URI + - POST /settings/autoFailover/resetCount +

      + Resets the number of nodes that Couchbase Server has automatically failed over. A request + can be sent to reset the auto-failover number to 0. This is a global setting, which applies to all + nodes in the cluster. Authentication is required to change this setting. No parameters are required. +

      + + POST /settings/autoFailover/resetCount
      -
      Syntax - curl -X POST -i -u [admin]:[password] \ - http://localhost:8091/settings/autoFailover/resetCount +
      + + + Syntax + + + curl -X POST -i -u [admin]:[password] \ + http://localhost:8091/settings/autoFailover/resetCount +
      -
      Example -

      Curl request example:

      +
      + + Example + + +

      + Curl request example: +

      - curl -X POST -i -u Administrator:password \ - http://10.5.2.54:8091/settings/autoFailover/resetCount + curl -X POST -i -u Administrator:password \ + http://10.5.2.54:8091/settings/autoFailover/resetCount -

      Raw HTTP request example:

      +

      + Raw HTTP request example: +

      - POST /settings/autoFailover/resetCount HTTP/1.1 + POST /settings/autoFailover/resetCount HTTP/1.1 Host: localhost:8091 Content-Type: application/x-www-form-urlencoded Authorization: Basic YWRtaW46YWRtaW4=
      -
      Response codes - HTTP/1.1 200 OK -

      Possible errors include:

      - This endpoint isn't available yet. +
      + + Response codes + + + HTTP/1.1 200 OK + +

      + Possible errors include: +

      + This endpoint isn't available yet. 401 Unauthorized
      diff --git a/content/rest-api/rest-cluster-autofailover-settings.dita b/content/rest-api/rest-cluster-autofailover-settings.dita index 83a6df06ea..d5e444c29e 100644 --- a/content/rest-api/rest-cluster-autofailover-settings.dita +++ b/content/rest-api/rest-cluster-autofailover-settings.dita @@ -1,61 +1,126 @@ - Retrieving Auto-Failover Settings - Auto-failover setting are retrieved with the GET /settins/autoFailover HTTP method and URI. + + Retrieving Auto-Failover Settings + + + Auto-failover settings are retrieved by means of the GET /settings/autoFailover HTTP method and URI. -
      HTTP method and URI -

      Use this request to retrieve any auto-failover settings for a cluster. Auto-failover is a - global setting for all clusters. Authenticated is required to read this value.

      - GET /settings/autoFailover +
      + + HTTP method and URI + -

      Parameters include:

      +

      + Retrieves auto-failover settings for a cluster. Auto-failover is a + global setting, which applies to all nodes in the cluster. To read auto-failover settings, + appropriate privileges are required. +

      + + GET /settings/autoFailover + +

      + Parameters include: +

      + +

      + +

        -
      • enabled: Value is true or false. True if auto-failover is enabled; - False if it is not.
        • -
      • timeout: Seconds that must elapse before auto-failover executes on a - cluster.
        • -
      • count [0|1]: Value is 0 or 1. It shows whether any node in a cluster can be - automatically failed-over.

        After one auto-failover occurs, the count is set to 1 and - Couchbase server does not perform auto-failover for the cluster again unless the count - is reset to 0. To failover more than one node at a time in a cluster, perform a manual - failover.

        • + +
      • + enabled. Indicates whether automatic failover is enabled (a value of true) or + disabled (a value of false). +
        • + +
      • + timeout. Returns an integer between 5 and 3600, which specifies the number of seconds set + to elapse, + after a node has become unavailable, before automatic failover is triggered. +
        • + +
      • + count. This parameter represents how many sequential auto-failover events have + occurred since the parameter + was itself last reset, to a value of 0, through administrator intervention. + The parameter's default value is 0. + Couchbase Server increments this value by + 1 for every auto-failover event that occurs, up to the maximum count — an administrator-specified number whose + highest-permitted value is 3. If sequential auto-failover events occur + until the maximum count is reached, no further + auto-failover is triggered until a parameter-reset is performed. +
        • +
      -
      Syntax -

      Curl request syntax:

      - curl -u [admin]:[password] http://[localhost]:8091/settings/autoFailover +
      + + Syntax + + +

      + Curl request syntax: +

      + + curl -u [admin]:[password] http://[localhost]:8091/settings/autoFailover
      -
      Example -

      Curl request example

      - curl -u Administrator:password http://10.5.2.54:8091/settings/autoFailover +
      + Example + + +

      + Curl request example: +

      + + curl -u Administrator:password http://10.5.2.54:8091/settings/autoFailover
      -
      Response -

      If successful Couchbase Server returns any auto-failover settings for the cluster:

      - { - "count": 0, - "enabled": false, - "timeout": 120 +
      + + Response + + +

      + If successful, execution returns the auto-failover settings for the cluster. For example: +

      + + { + "enabled": true, + "timeout": 72, + "count": 0, + "failoverOnDataDiskIssues": { + "enabled": true, + "timePeriod": 89 + }, + "failoverServerGroup": true, + "maxCount": 2 }
      -
      Response codes -

      Possible errors include:

      - HTTP/1.1 401 Unauthorized +
      + + Response codes + + +

      + Possible errors include: +

      + + HTTP/1.1 401 Unauthorized This endpoint isn't available yet. - GET /settings/autoFailover HTTP/1.1 + GET /settings/autoFailover HTTP/1.1 Host: localhost:8091 Authorization: Basic YWRtaW46YWRtaW4= Accept: */* - HTTP/1.1 200 OK + HTTP/1.1 200 OK Content-Type: application/json Content-Length: nnn { "enabled": false, "timeout": 30, "count": 0 } diff --git a/content/rest-api/rest-cluster-disable-query.dita b/content/rest-api/rest-cluster-disable-query.dita index eec3f3e00a..0ffe5753bc 100644 --- a/content/rest-api/rest-cluster-disable-query.dita +++ b/content/rest-api/rest-cluster-disable-query.dita @@ -1,13 +1,13 @@ - Disabling Consistent Query Results on Rebalance - Ensuring query results consistency is performed with the POST /internalSettings -d indexAwareRebalanceDisabled HTTP method, URI, and parameter. + Disabling Consistent View Query Results on Rebalance + Ensuring view query results consistency is performed with the POST /internalSettings -d indexAwareRebalanceDisabled HTTP method, URI, and parameter.
      Description -

      If queries are performed during rebalance, this setting ensures that query results are +

      If view queries are performed during rebalance, this setting ensures that query results are consistent with the original bucket and data organization prior to rebalancing. In other words, the query results reflect the data on an original node prior to rebalance rather than data on a node after rebalance started. By default, this functionality is enabled.

      diff --git a/content/rest-api/rest-cluster-email-notifications.dita b/content/rest-api/rest-cluster-email-notifications.dita index 0816c39991..4146fe9e17 100644 --- a/content/rest-api/rest-cluster-email-notifications.dita +++ b/content/rest-api/rest-cluster-email-notifications.dita @@ -9,6 +9,9 @@ Getting Alert Settings +
      + Email alerts can be dispatched automatically when a significant error occurs — see . +
      HTTP method and URI GET /settings/alerts
      diff --git a/content/rest-api/rest-cluster-intro.dita b/content/rest-api/rest-cluster-intro.dita index b4468aae57..0b6f35db47 100644 --- a/content/rest-api/rest-cluster-intro.dita +++ b/content/rest-api/rest-cluster-intro.dita @@ -57,7 +57,7 @@ GET, POST, PUT, DELETE /pools/default/serverGroups - Manages rack zone awareness (server groups). + Manages Server Group Awareness. Full diff --git a/content/rest-api/rest-endpoints-all.dita b/content/rest-api/rest-endpoints-all.dita index 2186f8c77c..b5dcb6d257 100644 --- a/content/rest-api/rest-endpoints-all.dita +++ b/content/rest-api/rest-endpoints-all.dita @@ -47,7 +47,7 @@ GET, POST, PUT, DELETE /pools/default/serverGroups - Manages rack zone awareness (server groups). + Manages Server Group Awareness (server groups). POST @@ -139,6 +139,11 @@ /settings/web Sets user names and passwords. + + POST + /controller/changePassword + Changes password for a given user. + POST /pools/default/memoryQuota diff --git a/content/rest-api/rest-fts-indexing.dita b/content/rest-api/rest-fts-indexing.dita index 4b2ccafbf4..403ee2bc97 100644 --- a/content/rest-api/rest-fts-indexing.dita +++ b/content/rest-api/rest-fts-indexing.dita @@ -26,7 +26,7 @@ "sourceParams": "", "sourceType": "nil", "sourceUUID": "", - "type": "bleve", + "type": "fulltext-index", "uuid": "6cc599ab7a85bf3b" } }, @@ -54,9 +54,10 @@ }, "sourceName": "", "sourceParams": "", - "sourceType": "nil", + "source + ": "nil", "sourceUUID": "", - "type": "bleve", + "type": "fulltext-index", "uuid": "6cc599ab7a85bf3b" }, "planPIndexes": [ diff --git a/content/rest-api/rest-fts.dita b/content/rest-api/rest-fts.dita index df64b2698a..cc7f91b00f 100644 --- a/content/rest-api/rest-fts.dita +++ b/content/rest-api/rest-fts.dita @@ -9,7 +9,7 @@ using SSL. If running in a multi-node environment, you have to connect to a node with Full Text service enabled.

      For more information on required roles and permissions, please consult . Granting users the appropriate + href="../security/security-roles.dita"/>. Granting users the appropriate role automatically assigns the required permissions, which are listed here for information purposes since you may see them in the audit log, for example.

      Indexing diff --git a/content/rest-api/rest-index-service.dita b/content/rest-api/rest-index-service.dita index 34239d86e1..6b63ea540d 100644 --- a/content/rest-api/rest-index-service.dita +++ b/content/rest-api/rest-index-service.dita @@ -5,11 +5,18 @@ The Index service REST API provides configuration options for the Index service.
      Description -

      Couchbase Server has data, index, and query services that can be enabled. - All of services can be enabled on a Couchbase Server node or a single service can be enabled on a Couchbase Server node. - Each node is as a member of a cluster.

      +

      + Couchbase Server provides a Data, an Index, and a Query Service. Each service + can be deployed and run either + with one or more of the others, on a single node; or on its own node. +

      +
      - Index service endpoints + + + Index-Service Endpoints + + @@ -46,12 +53,29 @@ stableSnapshotInterval. Full, Cluster + POST /settings/indexes Sets the parameter settings for the Index service. Full, Cluster + + + GET + /getIndexStatus + Retrieves status on defined indexes. + Full, Cluster + + + + GET + /getIndexStatement + Retrieves a statement on defined indexes. + Full, Cluster + + +
      diff --git a/content/rest-api/rest-node-failover.dita b/content/rest-api/rest-node-failover.dita index 5eec3e2aa0..ca88f5978b 100644 --- a/content/rest-api/rest-node-failover.dita +++ b/content/rest-api/rest-node-failover.dita @@ -12,33 +12,41 @@ replicated data on another node should be made available to clients. This endpoint along with the otpNode parameter (internal node name) fails over a specific node.

      - POST /controller/failOver + POST /controller/failOver
      Syntax

      HTTP request syntax:

      - POST /controller/failOver HTTP/1.1 + POST /controller/failOver HTTP/1.1 Authorization: Basic

      Curl request syntax:

      - curl -v -X POST + curl -v -X POST -u admin:password http://localhost:port/controller/failOver -d otpNode=[node@hostname]
      - Example + Examples - -

      The following examples fails over server node 10.3.3.63 from a cluster - with the local host, 10.5.2.54.

      +

      + The following curl statement performs a hard failover of server node 10.142.180.102 from a cluster + with localhost 10.142.180.1. +

      - curl -v -X POST \ - -u Administrator:password http://10.5.2.54:8091/controller/failOver \ - -d 'otpNode=ns_2@10.3.3.63' -
      + curl -v -X POST -u Administrator:password http://10.142.180.1:8091/controller/failOver \ +-d 'otpNode=ns_1@10.142.180.102' +

      + The following curl statement performs a hard failover of server nodes 10.142.180.102 and 10.142.180.103 from a + cluster with localhost 10.142.180.1. Note that multi-node failover is always hard + failover. +

      + curl -v -X POST -u Administrator:password http://10.142.180.1:8091/controller/failOver \ +-d 'otpNode=ns_1@10.142.180.102' -d 'otpNode=ns_1@10.142.180.103' + +
      Response codes @@ -70,11 +78,11 @@ Authorization: Basic

      The following example is a successful response:

      - HTTP/1.1 200 OK + HTTP/1.1 200 OK

      The following example is an unsuccessful response, for example, if the node does not exist in the cluster.

      - HTTP/1.1 400 + HTTP/1.1 400
      diff --git a/content/rest-api/rest-node-provisioning.dita b/content/rest-api/rest-node-provisioning.dita index a0bf4fc4f0..14086ad06f 100644 --- a/content/rest-api/rest-node-provisioning.dita +++ b/content/rest-api/rest-node-provisioning.dita @@ -33,9 +33,7 @@ curl -u Administrator:password -v -X POST http://[localhost]:8091/pools/default/ -d evictionPolicy=[valueOnly | full] -d ramQuotaMB=[value] -d bucketType=[membase | couchbase] - -d name=[bucket-name] - -d authType=[none | sasl] - -d saslPassword=[password]       + -d name=[bucket-name]
      Description

      Whether adding a node to an existing cluster or starting a new cluster, the node’s disk path @@ -88,8 +86,7 @@ curl -u Administrator:password -v -X POST http://192.168.42.101:8091/settings/w // Setup Bucket curl -u Administrator:password -v -X POST http://192.168.42.101:8091/pools/default/buckets \ -d 'flushEnabled=1&threadsNumber=3&replicaIndex=0&replicaNumber=0& \ - evictionPolicy=valueOnly&ramQuotaMB=597&bucketType=membase&name=default& \ - authType=sasl&saslPassword=password' + evictionPolicy=valueOnly&ramQuotaMB=597&bucketType=membase&name=default'

      Note: Service name for query service is different in REST APIs and CLI. REST API uses "n1ql" as the service name while CLI uses "query" in the services settings.

      diff --git a/content/rest-api/rest-node-services.dita b/content/rest-api/rest-node-services.dita index 0dce5120fb..d768d4ba20 100644 --- a/content/rest-api/rest-node-services.dita +++ b/content/rest-api/rest-node-services.dita @@ -10,14 +10,14 @@ curl -u [admin]:[password] [localhost]:8091/controller/addNode -d "hostname=[IPaddress]&user=[admin]&password=[password]" - -d "services=[data | index | n1ql | fts]" + -d "services=[data | index | n1ql | fts | cbas | eventing]"

      Joining a node to a cluster:

      curl -u [admin]:[password] -d clusterMemberHostIp=[ip-address] -d clusterMemberPort=[port] -d user=[admin] -d password=[password] http://[localhost]:8091/node/controller/doJoinCluster - -d "services=[data | index | n1ql | fts]" + -d "services=[data | index | n1ql | fts | cbas | eventing]"
      Description

      This command is used to add a node to a cluster and specify the service. If a service is not specified, the default (data) will be added.

      diff --git a/content/rest-api/rest-node-set-username.dita b/content/rest-api/rest-node-set-username.dita index 3a4c1c2052..6b6a6665a9 100644 --- a/content/rest-api/rest-node-set-username.dita +++ b/content/rest-api/rest-node-set-username.dita @@ -71,6 +71,11 @@ Cache-Control: no-cache no-store max-age=0
      +
      Changing Password +

      Changing the password is done via the /controller/changePassword + endpoint:curl -u <user>:<curr_password> http://<host>:8091/controller/changePassword -d password=<new_password>

      +

      For example:curl -u Administrator:password http://localhost:8091/controller/changePassword -d password=betterpasssword1

      +
      diff --git a/content/rest-api/rest-rza.dita b/content/rest-api/rest-rza.dita index 4376f3e478..1a2ab2bf0b 100644 --- a/content/rest-api/rest-rza.dita +++ b/content/rest-api/rest-rza.dita @@ -3,20 +3,18 @@ Server Groups API - The server groups REST API refers to the Rack Zone Awareness feature, which enables logical groupings - of servers on a cluster where each - server group physically belongs to a rack or availability zone. + The server groups REST API refers to the Server Group Awareness feature, which enables + logical groupings of servers on a cluster where each server group physically belongs to a rack + or availability zone. -
      Description +
      + The Server Group Awareness feature is only available in Couchbase + Server Enterprise Edition.Description

      This feature provides the ability to specify that active and corresponding replica partitions - be created on servers that are part of a separate rack or zone. For purposes of the server - group REST API, racks or availability zones are represented as flat space of server groups - with group names. To enable Rack Awareness, all servers in a cluster must be upgraded to use - the Rack Awareness feature.

      - - The Rack Awareness feature with its server group capability is an Enterprise - Edition feature. + be created on servers that are part of a separate rack or zone. For purposes of the server + group REST API, racks or availability zones are represented as flat space of server groups + with group names.

      The Server groups REST API provides the following capability:

      • Create server groups
        • diff --git a/content/rest-api/rest-security.dita b/content/rest-api/rest-security.dita index 8a5724a905..6232c7fe3f 100644 --- a/content/rest-api/rest-security.dita +++ b/content/rest-api/rest-security.dita @@ -80,6 +80,6 @@ - + diff --git a/content/rest-api/rest-server-nodes.dita b/content/rest-api/rest-server-nodes.dita index 89ab23e708..4d0c966969 100644 --- a/content/rest-api/rest-server-nodes.dita +++ b/content/rest-api/rest-server-nodes.dita @@ -57,6 +57,18 @@ Sets user names and passwords. Full + + POST + /controller/changePassword + Changes password for a given user. + Any + + + POST + /controller/changePassword + Changes password for a given user. + Any + POST /pools/default memoryQuota diff --git a/content/rest-api/rest-user-create.dita b/content/rest-api/rest-user-create.dita index 666acc3dec..fc78c32120 100644 --- a/content/rest-api/rest-user-create.dita +++ b/content/rest-api/rest-user-create.dita @@ -69,6 +69,6 @@ - + diff --git a/content/rest-api/rest-user-delete.dita b/content/rest-api/rest-user-delete.dita index c6ba940646..405a307244 100644 --- a/content/rest-api/rest-user-delete.dita +++ b/content/rest-api/rest-user-delete.dita @@ -36,6 +36,6 @@ - + diff --git a/content/rest-api/rest-user-getname.dita b/content/rest-api/rest-user-getname.dita index 7ddfc45a26..a9c678a4d1 100644 --- a/content/rest-api/rest-user-getname.dita +++ b/content/rest-api/rest-user-getname.dita @@ -39,6 +39,6 @@ - + diff --git a/content/rest-api/rest-user-password-put.dita b/content/rest-api/rest-user-password-put.dita index 8c94133013..b103d251a8 100644 --- a/content/rest-api/rest-user-password-put.dita +++ b/content/rest-api/rest-user-password-put.dita @@ -49,6 +49,6 @@ - + diff --git a/content/rest-api/rest-users.dita b/content/rest-api/rest-users.dita index b53f8d8ab7..1cacfe8dc4 100644 --- a/content/rest-api/rest-users.dita +++ b/content/rest-api/rest-users.dita @@ -58,6 +58,6 @@ - + diff --git a/content/rest-api/rest-views-intro.dita b/content/rest-api/rest-views-intro.dita index 327d4fe260..345544e92e 100644 --- a/content/rest-api/rest-views-intro.dita +++ b/content/rest-api/rest-views-intro.dita @@ -53,7 +53,7 @@ PUT /[bucket_name]/_design/[ddoc-name] - Creates a news design document with one or more views. + Creates a new design document with one or more views. Full, Cluster diff --git a/content/rest-api/rest-x509.dita b/content/rest-api/rest-x509.dita index c3adbae50f..db88820916 100644 --- a/content/rest-api/rest-x509.dita +++ b/content/rest-api/rest-x509.dita @@ -54,6 +54,6 @@ - + diff --git a/content/rest-api/rest-xdcr-create-replication.dita b/content/rest-api/rest-xdcr-create-replication.dita index 404b61e0c6..a4f3ca6933 100644 --- a/content/rest-api/rest-xdcr-create-replication.dita +++ b/content/rest-api/rest-xdcr-create-replication.dita @@ -23,7 +23,8 @@ -d toCluster=[cluster-name] -d toBucket=[bucket-name] -d replicationType=continuous - -d type=[capi | xmem] + -d type=[capi | xmem] + -d compressionType=["None" | "Auto" | "Snappy"] The type values, capi and xmem, are represented by version1 and version2 in the web console. Default: xmem. @@ -33,14 +34,17 @@
      Example -

      Curl request example:

      - curl -v -X POST -u Administrator:password1 \ - http://10.4.2.4:8091/controller/createReplication \ - -d 'fromBucket=beer-sample' \ - -d 'toCluster=remote1' \ - -d 'toBucket=remote_beer' \ - -d 'replicationType=continuous' \ - -d 'type=capi' +

      The following example creates a replication between the source and remote + buckets. The remote bucket is located on a cluster named 10.142.171.101. + Note that data compression is enabled, by specifying the value 1 (specifying 0 + disables compression).

      + curl -v -X POST -u Administrator:password \ +http://127.0.0.1:8091/controller/createReplication \ +-d fromBucket=source \ +-d toCluster=10.142.171.101 \ +-d toBucket=remote \ +-d replicationType=continuous \ +-d compressionType="Snappy"

      Raw HTTP request:

      POST / HTTP/1.1 diff --git a/content/rest-api/rest-xdcr-data-encrypt.dita b/content/rest-api/rest-xdcr-data-encrypt.dita index 4fc0a5dfe8..a82760e882 100644 --- a/content/rest-api/rest-xdcr-data-encrypt.dita +++ b/content/rest-api/rest-xdcr-data-encrypt.dita @@ -55,30 +55,82 @@ following HTTP method and URI:

      HTTP method and URI

      - GET /pools/default/certificate + GET /pools/default/certificate

      Syntax

      - curl http://[remoteHost]:[port]/pools/default/certificate + curl http://[remoteHost]:[port]/pools/default/certificate

      Example

      - curl http://remoteHost:8091/pools/default/certificate > ./remoteCert.pem + curl http://remoteHost:8091/pools/default/certificate > ./remoteCert.pem
      Regenerating certificates

      To regenerate a certificate on a destination cluster, use the following HTTP method and URI:

      HTTP method and URI

      - POST /controller/regenerateCertificate + POST /controller/regenerateCertificate

      Example

      - curl -X POST http://Administrator:asdasd@remoteHost:8091/controller/regenerateCertificate + curl -X POST http://Administrator:asdasd@remoteHost:8091/controller/regenerateCertificate
      -
      Configuring XDCR with data encryption -

      A POST to /pools/default/remoteClusters creates the XDCR cluster reference from the - source cluster to the destination cluster. Setting the demandEncryption parameter to - one (1) and providing the certificate name and location enables data encryption.

      +
      + + + Configuring XDCR with data encryption + + +

      + A POST to /pools/default/remoteClusters creates the XDCR cluster reference from the + source cluster to the destination cluster. The secureType parameter can be: +

      + +
        +
      • + none: The connection is not secured. + +

        + +

        +
        • + +
      • + half: Only the password is encrypted. If the destination cluster is running Couchbase + Enterprise Server verion 5.5 or later, SCRAM-SHA is used, and no certificate is required. If the + destination cluster is running a pre-5.5 version of Couchbase Enterprise Server, TLS is used, and + the root certificate of the destination cluster must be provided. In both cases, the username and + password of the destination cluster's Full, Cluster, or XDCR Administrator must be provided. If + specified, the root certificate of the destination cluster is ignored if the destination cluster is + running a pre-5.5 version of Couchbase Enterprise Edition. + +

        + +

        +
        • + +
      • + full: Both password and data are encrypted. TLS is used. This requires either: + +
          +
        • The username and password of the destination cluster's Full, Cluster, or XDCR Administrator; + plus the root certificate of the destination cluster. + +

          + +

          +
          • + +
        • + The root certificate of the destination cluster; plus the client certificate and client private + key of the local cluster. +

          + +

          +
          • +
        +
        • +

      HTTP method and URI

      The following HTTP method and URI modifies the destination cluster reference.

      @@ -87,23 +139,80 @@

      Syntax

      - curl -X POST -u Admin:myPassword + curl -X POST -u Admin:myPassword http://localHost:port/pools/default/remoteClusters -d name=<clusterName> // Remote cluster name -d hostname=<host>:<port> // FQDN of the remote host. -d username=<adminName> // Remote cluster Admin name -d password=<adminPassword> // Remote cluster Admin password - -d demandEncryption=[0|1] --data-urlencode "certificate=$(cat remoteCert.pem)" - + -d encryptionType=["half"|"full"] --data-urlencode "certificate=$(cat remoteCert.pem)" + // set to "half" to just encrypt log-in details -

      Example

      +

      Example 1

      + +

      + This secures password and data, by means of TLS. +

      - curl -X POST - -d 'name=remoteName' \ - -d 'hostname=10.3.4.187:8091' \ - -d 'username=remoteAdmin' -d 'password=remotePassword' \ - -d 'demandEncryption=1 --data-urlencode "certificate=$(cat remoteCert.pem)"' \ - http://Administrator:asdasd@192.168.0.1:8091/pools/default/remoteClusters/
      + curl -u Administrator:password -v -X POST http://10.142.180.1:8091/pools/default/remoteClusters +-d 'name=remoteName' \ +-d 'hostname=10.142.180.101:8091' \ +-d 'username=remoteAdministrator' \ +-d 'password=remotePassword' \ +-d 'encryptionType=full' \ +--data-urlencode "certificate=$(cat ./cert/remoteClusterRootCert.pem)" + +

      + Example 2 +

      + +

      + This secures both authentication and data, by means of TLS; using root and client certificates, and + client private key. +

      + + curl -X POST -u Administrator:welcome \ +http://127.0.0.1:9000/pools/default/remoteClusters \ +-d name=rem \ +-d hostname=127.0.0.1:9001 \ +-d secureType=full \ +--data-urlencode "certificate=$(cat /Users/yu/cert/root/ca.pem)" \ +--data-urlencode "clientCertificate=$(cat /Users/yu/pem/chain.pem)" \ +--data-urlencode "clientKey=$(cat /Users/yu/cert/client/client/client.key)" + +

      + Example 3 +

      + +

      + This secures password only, by means of SCRAM-SHA. +

      + + curl -u Administrator:password -v -X POST http://10.142.180.1:8091/pools/default/remoteClusters \ +-d 'name=remoteName' \ +-d 'hostname=10.142.180.101:8091' \ +-d 'username=remoteAdministrator' \ +-d 'password=remotePassword' \ +-d 'secureType=half' + +

      + The response obtained from successfully securing the password with SCRAM-SHA is as follows: +

      + + { +"deleted": false, +"demandEncryption": true, +"encryptionType": "half", +"hostname": "10.142.180.101:8091", +"name": "remoteName", +"secureType": "half", +"uri": "/pools/default/remoteClusters/remoteName", +"username": "remoteAdministrator", +"uuid": "9b323738eb48104a72d15e7230911358", +"validateURI": "/pools/default/remoteClusters/remoteName?just_validate=1" +} + +
      Disabling data encryption @@ -116,7 +225,7 @@ PUT /pools/default/remoteClusters

      Example

      - curl -X PUT -u myAdmin:myPassword + curl -X PUT -u myAdmin:myPassword http://192.168.0.1:8091/pools/default/remoteClusters/ \ -d 'name=remoteName' \ -d 'hostname=10.3.4.187:8091'\ diff --git a/content/rest-api/rest-xdcr-statistics.dita b/content/rest-api/rest-xdcr-statistics.dita index 652d7abfad..2fa8b87128 100644 --- a/content/rest-api/rest-xdcr-statistics.dita +++ b/content/rest-api/rest-xdcr-statistics.dita @@ -20,18 +20,18 @@
      HTTP method and URI

      The destination endpoint follows the - /pools/default/buckets/@xdcr_[bucket_name]/stats/ URI endpoint:

      - GET /pools/default/buckets/@xdcr_[bucket_name]/stats/[destination_endpoint] + /pools/default/buckets/@xdcr-[source_bucket]/stats/ URI endpoint:

      + GET /pools/default/buckets/@xdcr-[source_bucket]/stats/[destination_endpoint]

      Where the destination endpoint is:

      replications/[remote_UUID]/[source_bucket]/[destination_bucket]/[stat_name]

      Where the HTTP endpoint string with full URI is:

      - http://[localhost]:[port]/pools/default/buckets/[bucket_name]/stats/replications/ \ + http://[node address]:[port]/pools/@xdcr-[source_bucket]/buckets/[bucket_name]/stats/replications/ \ [remote_UUID]/[source_bucket]/[destination_bucket]/[stat_name]

      Where the HTTP string with a properly URL-encoded URI is:

      - http://[localhost]:[port]/pools/default/buckets/[bucket_name]/stats/ \ + http://[node address]:[port]/pools/@xdcr-[source_bucket]/buckets/[bucket_name]/stats/ \ replications%2F[remote_UUID]%2F[source_bucket]%2F[destination_bucket]%2F[stat_name] @@ -189,7 +189,7 @@ curl -u Administrator:password http://10.5.2.54:8091/pools/default/remoteCluster Retrieving <codeph>docs_written</codeph> stats
      HTTP method and URI - GET /pools/default/buckets/@xdcr_[bucket_name]/stats/[destination_endpoint] + GET /pools/default/buckets/@xdcr-[source_bucket]/stats/[destination_endpoint]

      Where the [destination_endpoint] is:

      replications/[remote_UUID]/[source_bucket]/[destination_bucket]/docs_written @@ -199,15 +199,15 @@ curl -u Administrator:password http://10.5.2.54:8091/pools/default/remoteCluster
      Syntax

      Curl request syntax for number of documents written:

      curl -u [admin]:[password] - http://[localhost]:8091/pools/default/buckets/default/stats/ + http://[node address]:8091/pools/default/buckets/@xdcr-[source_bucket]/stats/ replications%2F[remote_UUID]%2F[source_bucket]%2F[destination_bucket]%2Fdocs_written
      Example

      To get the number of documents written:

      curl -u Administrator:password \ - http://10.5.2.54:8091/pools/default/buckets/default/stats/ \ - replications%2F8ba6870d88cd72b3f1db113fc8aee675%2Fsource_bucket%2Fdestination_bucket%2Fdocs_written + http://10.5.2.54:8091/pools/default/buckets/@xdcr-default/stats/ \ + replications%2F8ba6870d88cd72b3f1db113fc8aee675%2Fdefault%2Fdefault%2Fdocs_written
      @@ -237,7 +237,7 @@ curl -u Administrator:password http://10.5.2.54:8091/pools/default/remoteCluster
      HTTP method and URI - GET /pools/default/buckets/@xdcr_[bucket_name]/stats/[destination_endpoint] + GET /pools/default/buckets/@xdcr-[source_bucket]/stats/[destination_endpoint]

      Where the [destination_endpoint] is:

      replications/[remote_UUID]/[source_bucket]/[destination_bucket]/rate_replication @@ -247,7 +247,7 @@ curl -u Administrator:password http://10.5.2.54:8091/pools/default/remoteCluster
      Syntax

      Curl request syntax:

      curl -u [admin]:[password] - http://[localhost]:8091/pools/default/buckets/default/stats/ + http://[node address]:8091/pools/default/buckets/[source_bucket]/stats/ replications%2F[remote_UUID]%2F[source_bucket]%2F[destination_bucket]%2Frate_replication
      @@ -255,8 +255,8 @@ curl -u Administrator:password http://10.5.2.54:8091/pools/default/remoteCluster
      Example

      Curl request example to get the rate of replication:

      curl -u Administrator:password \ - http://10.5.2.54:8091/pools/default/buckets/default/stats/ \ - replications%2F8ba6870d88cd72b3f1db113fc8aee675%2Fsource_bucket%2Fdestination_bucket%2Frate_replication + http://10.5.2.54:8091/pools/default/buckets/@xdcr-default/stats/ \ + replications%2F8ba6870d88cd72b3f1db113fc8aee675%2Fdefault%2Fdefault%2Frate_replication
      @@ -282,7 +282,7 @@ curl -u Administrator:password http://10.5.2.54:8091/pools/default/remoteCluster
      HTTP method and URI - GET /pools/default/buckets/@xdcr_[bucket_name]/stats/[destination_endpoint] + GET /pools/default/buckets/@xdcr-[source_bucket]/stats/[destination_endpoint]

      Where the [destination_endpoint] is:

      replications/[remote_UUID]/[source_bucket]/[destination_bucket]/docs_opt_repd @@ -291,7 +291,7 @@ curl -u Administrator:password http://10.5.2.54:8091/pools/default/remoteCluster
      Syntax: get replication id curl -s -u admin:password \ - http://[localhost]:8091/pools/default/tasks + http://[node address]:8091/pools/default/tasks
      Example: get replication id @@ -316,8 +316,8 @@ curl -u Administrator:password http://10.5.2.54:8091/pools/default/remoteCluster

      With this replication id, retrieve a sampling of stats for docs_opt_repd:

      curl -s -u Administrator:password \ -http://10.3.121.119:8091/pools/default/buckets/default/stats/ \ -replications%2fdef03dbf5e968a47309194ebe052ed21%2fbucket_source%2fbucket_destination%2fdocs_opt_repd +http://10.3.121.119:8091/pools/default/buckets/@xdcr-default/stats/ \ +replications%2fdef03dbf5e968a47309194ebe052ed21%2fdefault%2fdefault%2fdocs_opt_repd
      Response @@ -349,7 +349,7 @@ replications%2fdef03dbf5e968a47309194ebe052ed21%2fbucket_source%2fbucket_destina

      Curl request syntax:

      curl -u [admin]:[password] -X GET - http://[Destination_IP]:8091/pools/default/buckets/[bucket_name]/stats + http://[Destination_IP]:8091/pools/default/buckets/[destination bucket]/stats
      Example diff --git a/content/sdk/c.ditamap b/content/sdk/c.ditamap index 0b002a7e9f..d3c4678c68 100644 --- a/content/sdk/c.ditamap +++ b/content/sdk/c.ditamap @@ -2,11 +2,11 @@ C SDK Guide - + - 2.8.3 + 2.8.5 @@ -27,6 +27,9 @@ + + + @@ -35,7 +38,9 @@ - + + + @@ -47,6 +52,7 @@ + @@ -58,5 +64,7 @@ + + diff --git a/content/sdk/c/compatibility-versions-features.dita b/content/sdk/c/compatibility-versions-features.dita index d22a089785..dc74636855 100644 --- a/content/sdk/c/compatibility-versions-features.dita +++ b/content/sdk/c/compatibility-versions-features.dita @@ -84,7 +84,64 @@ lcb_sched_leave(instance);

    - Feature Compatibility + Couchbase Version/SDK Version Matrix +

    Couchbase SDKs are tested against a variety of different environments to ensure both + backward and forward compatibility with different versions of Couchbase Server. The matrix + below denotes the version of Couchbase Server, the version of the C SDK client, and whether the SDK + is:

      +
    • ✖ Unsupported: This combination is not tested, and is not within the scope of + technical support if you have purchased a support agreement.
      • +
    • ◎ Compatible: This combination has been tested previously, and should be + compatible. This combination is not recommended by our technical support + organization. It is best to upgrade either the SDK or the Couchbase version you are + using.
      • +
    • ✔ Supported:This combination is subject to ongoing quality assurance, and is + fully supported by our technical support organization.
      • +

    + + Recommended SDK per Server Version Matrix + + + + + + + + + + SDK 2.5 + SDK 2.6 + SDK 2.7 + SDK 2.8 + + + + + + Server 3.1 + ✔ + ✔ ✔ + ✔ + + + Server 4.0-4.6 + ✔ + ✔ ✔ + ✔ + + + Server 5.0-5.1 + ✔ + ✔ ✔ + ✔ + + + +
    +

    Note the + End of Life dates for Couchbase Server and SDK versions. + See the notes there for Support details.

    +

    To take advantage of all features offered by Couchbase Server, you need to know what version of the client provides compatibility for the features you want to use. The following matrix shows which versions of the C SDK client support the major features diff --git a/content/sdk/c/encrypting-using-sdk.dita b/content/sdk/c/encrypting-using-sdk.dita new file mode 100644 index 0000000000..ed69079f51 --- /dev/null +++ b/content/sdk/c/encrypting-using-sdk.dita @@ -0,0 +1,105 @@ + + + + Field Level Encryption from the C SDK + Field Level Encryption is available in Couchbase Data Platform 5.5, from C SDK (Libcouchbase) version 2.8.6. + +

    +

    + + +
    + Test Sample +

    Here is a brief example of the code to encrypt a field using libcouchbase, the C SDK..

    +
    + + +
    + Sample Code +

    static void store_encrypted(lcb_t instance, const char *key, const char *val) +{ + lcb_error_t err; + lcb_CMDSTORE cmd = {}; + lcbcrypto_CMDENCRYPT ecmd = {}; + lcbcrypto_FIELDSPEC field = {}; + + printf("KEY: %s\n", key); + printf("PLAIN: %s\n", val); + + ecmd.version = 0; + ecmd.prefix = NULL; + ecmd.doc = val; + ecmd.ndoc = strlen(val); + ecmd.out = NULL; + ecmd.nout = 0; + ecmd.nfields = 1; + ecmd.fields = &field; + field.name = "message"; + field.alg = "AES-256-HMAC-SHA256"; + field.kid = "mypublickey"; + + err = lcbcrypto_encrypt_document(instance, &ecmd); + if (err != LCB_SUCCESS) { + die(instance, "Couldn't encrypt field 'message'", err); + } + printf("CIPHER: %s\n", ecmd.out); + + LCB_CMD_SET_KEY(&cmd, key, strlen(key)); + LCB_CMD_SET_VALUE(&cmd, ecmd.out, ecmd.nout); + cmd.operation = LCB_SET; + cmd.datatype = LCB_DATATYPE_JSON; + + err = lcb_store3(instance, NULL, &cmd); + free(ecmd.out); // NOTE: it should be compatible with what providers use to allocate memory + if (err != LCB_SUCCESS) { + die(instance, "Couldn't schedule storage operation", err); + } + lcb_wait(instance); +}

    + +

    To decrypt the data again, use the following:static void get_callback(lcb_t instance, int cbtype, const lcb_RESPBASE *rb) +{ + if (rb->rc == LCB_SUCCESS) { + const lcb_RESPGET *rg = (const lcb_RESPGET *)rb; + lcbcrypto_CMDDECRYPT dcmd = {}; + lcb_error_t err; + + printf("VALUE: %.*s\n", (int)rg->nvalue, rg->value); + dcmd.version = 0; + dcmd.prefix = NULL; + dcmd.doc = rg->value; + dcmd.ndoc = rg->nvalue; + dcmd.out = NULL; + dcmd.nout = 0; + err = lcbcrypto_decrypt_document(instance, &dcmd); + if (err != LCB_SUCCESS) { + die(instance, "Couldn't decrypt field 'message'", err); + } + if (dcmd.out == NULL) { + die(instance, "Crypto provider returned success, but document is NULL", LCB_EINVAL); + } + printf("PLAIN: %.*s\n", (int)dcmd.nout, dcmd.out); + free(dcmd.out); // NOTE: it should be compatible with what providers use to allocate memory + printf("CAS: 0x%" PRIx64 "\n", rb->cas); + } else { + die(instance, lcb_strcbtype(cbtype), rb->rc); + } +} + +static void get_encrypted(lcb_t instance, const char *key) +{ + lcb_CMDGET cmd = {}; + lcb_error_t err; + LCB_CMD_SET_KEY(&cmd, key, strlen(key)); + printf("KEY: %s\n", key); + err = lcb_get3(instance, NULL, &cmd); + if (err != LCB_SUCCESS) { + die(instance, "Couldn't schedule get operation", err); + } + lcb_wait(instance); +}

    +
    + + + diff --git a/content/sdk/c/full-text-searching-with-sdk.dita b/content/sdk/c/full-text-searching-with-sdk.dita index bbc9af7d40..30f06509ac 100644 --- a/content/sdk/c/full-text-searching-with-sdk.dita +++ b/content/sdk/c/full-text-searching-with-sdk.dita @@ -43,6 +43,326 @@ href="../full-text-search-overview.dita#full-text-search/facets" >facets

    Refer to for more information - about the encoded query format.

    + about the encoded query format. A general introduction to FTS, with pointers to detailed descriptions of its + principal features, is provided in + Full Text Search: Fundamentals.

    + +
    Detailed Examples + +

    + The code example below demonstrates the Full Text Search + API. The example assumes that Couchbase Server is running, and that + the username Administrator and + the password password provide authorization for performing the searches. + It also assumes that the travel-sample bucket + has been installed. + For information on creating users and managing roles, see + Authorization. + For information on installing sample buckets, see + Settings. + Don't forget to create an lcb_CMDFTS structure, and populate it with a query, as in + the first example, above. +

    + +

    + The example also assumes the existence of three specific Full Text Indexes, defined on the + travel-sample bucket. These are: +

    + +
      +
    • + travel-sample-index-unstored: Uses only the default + settings. +
      • + +
    • + travel-sample-index-stored: Uses default settings, with one exception: + dynamic fields are stored, for the whole index. +
      • + +
    • + travel-sample-index-hotel-description: Indexes only the description + fields of hotel + documents, and disables the default type mapping. The index + has a custom analyzer named myUnicodeAnalyzer defined on it: the analyzer's + main characteristic is that it uses the unicode tokenizer. +
      • +
    + +

    + See + Creating Indexes + for details on how to create these indexes: they can be created interactively, by + means of the Couchbase Web Console; however, there may be greater efficiency + in using the Couchbase REST + API, as described + in the section + Index Creation with the REST API. + The JSON objects that constitute index-definitions (for + inclusion as bodies to the index-creation REST calls), are provided in + Demonstration Indexes. +

    + +

    + The example + example/fts/fts.c + features the following Full Text Searches on the + travel-sample bucket, within Couchbase Server: +

    + +
      + +
    • + Simple Text Query on a single word, targeting an index with dynamic fields unstored. + ----> {"indexName":"travel-sample-index-unstored","size":10,"highlight":{"style":"html"},"query":{"match":"swanky"}} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"landmark_25956","score":0.8037132228729517,"locations":{"content":{"swanky":[{"pos":26,"start":146,"end":152,"array_positions":null}]}},"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_54820232","id":"landmark_25888","score":0.7269537231028589,"locations":{"content":{"swanky":[{"pos":8,"start":52,"end":58,"array_positions":null}]}},"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_54820232","id":"landmark_3421","score":0.7066356080640858,"locations":{"content":{"swanky":[{"pos":23,"start":123,"end":129,"array_positions":null}]}},"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_54820232","id":"hotel_25598","score":0.3380266349466091,"locations":{"reviews.content":{"swanky":[{"pos":139,"start":711,"end":717,"array_positions":[0]}]}},"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_6ddbfb54","id":"hotel_25794","score":0.3060403881659807,"locations":{"reviews.content":{"swanky":[{"pos":23,"start":125,"end":131,"array_positions":[1]}]}},"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"hotel_26223","score":0.2889878306400093,"locations":{"description":{"swanky":[{"pos":1,"start":0,"end":6,"array_positions":null}]}},"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"hotel_16350","score":0.269900851046037,"locations":{"reviews.content":{"swanky":[{"pos":123,"start":714,"end":720,"array_positions":[3]}]}},"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"hotel_25800","score":0.255991517659089,"locations":{"reviews.content":{"swanky":[{"pos":15,"start":80,"end":86,"array_positions":[1]}]}},"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_54820232","id":"hotel_25301","score":0.22660209598987638,"locations":{"reviews.content":{"swanky":[{"pos":7,"start":25,"end":31,"array_positions":[3]}]}},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"match":"swanky","prefix_length":0,"fuzziness":0,"operator":"or"},"size":10,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":9,"max_score":0.8037132228729517,"took":209051,"facets":null} + +
      • + +
    • + Simple Text Query, specifying an index + that consists only of content derived from a specific field from a + specific document-type. + ----> {"indexName":"travel-sample-index-hotel-description","size":10,"query":{"match":"swanky"}} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"match":"swanky","prefix_length":0,"fuzziness":0,"operator":"or"},"size":10,"from":0,"highlight":null,"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":0,"max_score":0,"took":133187,"facets":null} + +
      • + +
    • + Simple Text Query on Stored Field, specifying the field to be searched; targeting an index with dynamic + fields stored, to ensure that field-content is included in the return object. + + ----> {"indexName":"travel-sample-index-stored","size":10,"highlight":{"style":"html"},"query":{"field":"destinationairport","match":"MDG"}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"route_30377","score":8.864611972602955,"locations":{"destinationairport":{"mdg":[{"pos":1,"start":0,"end":3,"array_positions":null}]}},"fragments":{"destinationairport":["\u003cmark\u003eMDG\u003c/mark\u003e"]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"match":"MDG","field":"destinationairport","prefix_length":0,"fuzziness":0,"operator":"or"},"size":10,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":1,"max_score":8.864611972602955,"took":139750,"facets":null} + + +
      • + +
    • + Match Query with Facet, showing how query-results can be displayed + either by row or by hits; + and demonstrating use of a facet, which provides aggregation-data. + ----> {"indexName":"travel-sample-index-stored","highlight":{"style":"html"},"size":10,"query":{"field":"reviews.content","match":"La Rue Saint Denis!!"},"facets":{"Countries Referenced":{"size":5,"field":"country"}}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_26223","score":1.1060123561061241,"locations":{"reviews.content":{"denis":[{"pos":88,"start":443,"end":448,"array_positions":[0]}],"la":[{"pos":85,"start":430,"end":432,"array_positions":[0]}],"rue":[{"pos":86,"start":433,"end":436,"array_positions":[0]}],"saint":[{"pos":87,"start":437,"end":442,"array_positions":[0]}]}},"fragments":{"reviews.content":["…libre had thieves? After all, this is supposed to be a 4 stars hotel, not a bedsit next to \u003cmark\u003eLa\u003c/mark\u003e \u003cmark\u003eRue\u003c/mark\u003e \u003cmark\u003eSaint\u003c/mark\u003e \u003cmark\u003eDenis\u003c/mark\u003e!!! I complained to the manager, Ms. Laura Lamblin, who told me that I had to go to the pol…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_8662","score":0.18089717282438075,"locations":{"reviews.content":{"rue":[{"pos":163,"start":943,"end":946,"array_positions":[3]}],"saint":[{"pos":168,"start":971,"end":976,"array_positions":[3]}]}},"fragments":{"reviews.content":["…nd within easy walking distance of the myriad of shops and restaurants on both the \u003cmark\u003eRue\u003c/mark\u003e de Rennes and Boulevard \u003cmark\u003eSaint\u003c/mark\u003e-Germain. The hotel serves a decent continental breakfast which seems expensive at 1…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_26489","score":0.11262421677946458,"locations":{"reviews.content":{"la":[{"pos":12,"start":55,"end":57,"array_positions":[0]}]}},"fragments":{"reviews.content":["…nt place to stay whilst in \u003cmark\u003eLA\u003c/mark\u003e. We were in a large suite and it was wonderful - the best bed and linen I have experienced in a hotel in a long time! I was travelling for work with my boyfriend and we b…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_37339","score":0.07801362501832362,"locations":{"reviews.content":{"la":[{"pos":76,"start":442,"end":444,"array_positions":[0]}]}},"fragments":{"reviews.content":["…e grounds and buildings were maintained to the highest standards. We got reservations in all the a \u003cmark\u003ela\u003c/mark\u003e carte restaurants and particularly enjoyed the Japenese ( we managed to get in there 3 times!!) an…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_35854","score":0.07486989518551089,"locations":{"reviews.content":{"saint":[{"pos":273,"start":1672,"end":1677,"array_positions":[0]}]}},"fragments":{"reviews.content":["…s much of the island, which makes one appreciate even more the privacy of Tamarindo Estates. We have stayed on Belize, Bonaire, \u003cmark\u003eSaint\u003c/mark\u003e Vincent, and Puerto Rico’s big island. Culebra tops all, and Tam…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_19324","score":0.07299937184634361,"locations":{"reviews.content":{"la":[{"pos":30,"start":169,"end":171,"array_positions":[1]},{"pos":32,"start":180,"end":182,"array_positions":[1]},{"pos":25,"start":127,"end":129,"array_positions":[2]}]}},"fragments":{"reviews.content":["…I am very satisfied with the hotel. Location is excellent for sightseeing - next to \u003cmark\u003eLa\u003c/mark\u003e Rambla, \u003cmark\u003eLa\u003c/mark\u003e Boqueria Market and Liceu subway station, few steps to Catalunia (where the airport bus stops). Though…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_19323","score":0.07179334544788325,"locations":{"reviews.content":{"la":[{"pos":32,"start":173,"end":175,"array_positions":[1]},{"pos":35,"start":188,"end":190,"array_positions":[1]},{"pos":28,"start":150,"end":152,"array_positions":[4]},{"pos":34,"start":179,"end":181,"array_positions":[4]},{"pos":76,"start":411,"end":413,"array_positions":[4]},{"pos":197,"start":1112,"end":1114,"array_positions":[4]},{"pos":28,"start":141,"end":143,"array_positions":[5]},{"pos":18,"start":97,"end":99,"array_positions":[6]}]}},"fragments":{"reviews.content":["…e in Barcelona. The Hotel Curious is perfectly located down a cozy street just off of \u003cmark\u003eLa\u003c/mark\u003e Ramblas and \u003cmark\u003eLa\u003c/mark\u003e Boqueria. You are a 10-minute walk from the Liceu (Green) and 15-minutes from Plaza Catalunya (a…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_26212","score":0.0659503730344147,"locations":{"reviews.content":{"la":[{"pos":28,"start":132,"end":134,"array_positions":[0]},{"pos":23,"start":118,"end":120,"array_positions":[1]},{"pos":32,"start":180,"end":182,"array_positions":[2]},{"pos":8,"start":33,"end":35,"array_positions":[4]},{"pos":56,"start":288,"end":290,"array_positions":[4]}]}},"fragments":{"reviews.content":["…e opportunity came up to spend a night at the newly built Shangri-\u003cmark\u003eLa\u003c/mark\u003e Hotel for New Years Eve, we couldn't resist. We had heard so many wonderful things and were very excited. Our good friend Jeffrey V…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_26498","score":0.06565206104519802,"locations":{"reviews.content":{"la":[{"pos":150,"start":861,"end":863,"array_positions":[1]},{"pos":182,"start":1025,"end":1027,"array_positions":[1]}]}},"fragments":{"reviews.content":["… but because our stays in \u003cmark\u003eLA\u003c/mark\u003e were just one-dayers this time, we chose an airport hotel, but got the bus into Santa Monica and West Hollywood for the day. Looking forward to our next stay in \u003cmark\u003eLA\u003c/mark\u003e at West…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_20737","score":0.0654032654874764,"locations":{"reviews.content":{"la":[{"pos":15,"start":63,"end":65,"array_positions":[0]},{"pos":20,"start":115,"end":117,"array_positions":[2]},{"pos":4,"start":9,"end":11,"array_positions":[3]},{"pos":23,"start":105,"end":107,"array_positions":[3]}]}},"fragments":{"reviews.content":["…ays since I was working at the \u003cmark\u003eLA\u003c/mark\u003e Auto Show at the convention center. Last year I stayed at the Westin (see review). So, I have to admit that this location is definitly MUCH better. Much more within a…"]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"match":"La Rue Saint Denis!!","field":"reviews.content","prefix_length":0,"fuzziness":0,"operator":"or"},"size":10,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":{"Countries Referenced":{"size":5,"field":"country"}},"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":102,"max_score":1.1060123561061241,"took":7221735,"facets":{"Countries Referenced":{"field":"country","total":192,"missing":0,"other":0,"terms":[{"term":"united","count":90},{"term":"kingdom","count":49},{"term":"states","count":41},{"term":"france","count":12}]}}} + +
      • + +
    • + DocId Query, showing results of a query on two document IDs. + ----> {"indexName":"travel-sample-index-unstored","query":{"ids":["hotel_26223","hotel_28960"]}} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_6ddbfb54","id":"hotel_28960","score":1,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"hotel_26223","score":1,"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"ids":["hotel_26223","hotel_28960"]},"size":10,"from":0,"highlight":null,"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":2,"max_score":1,"took":111182,"facets":null} + +
      • + +
    • + Unanalyzed Term Query with Fuzziness Level of 0, demonstrating how to query on a term with no analysis. + Zero fuzziness is specified, to ensure + that matches are exact. + ----> {"indexName":"travel-sample-index-stored","size":50,"highlight":{"style":"html"},"query":{"field":"reviews.content","term":"sushi"}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_16527","score":0.5115191736068728,"locations":{"reviews.content":{"sushi":[{"pos":168,"start":898,"end":903,"array_positions":[5]},{"pos":193,"start":1040,"end":1045,"array_positions":[5]}]}},"fragments":{"reviews.content":["…aff were superb. Sancho's \u003cmark\u003eSushi\u003c/mark\u003e Bar was the only disappointment - food was not good and prices were too high. Coming from San Francisco and Washington DC we have great \u003cmark\u003esushi\u003c/mark\u003e restaurants. Sancho's coul…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_23198","score":0.45801365906648533,"locations":{"reviews.content":{"sushi":[{"pos":209,"start":1121,"end":1126,"array_positions":[0]},{"pos":215,"start":1150,"end":1155,"array_positions":[0]}]}},"fragments":{"reviews.content":["… Encore/Wynn. Starbucks for breakfast is half the price of the cafe in the hotel. Also, \u003cmark\u003eSushi\u003c/mark\u003e Ra has good half price \u003cmark\u003esushi\u003c/mark\u003e from 5 - 7pm. (4) Boarding pass: Encore/Wynn lets you print it for free in th…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_21844","score":0.40346491995392636,"locations":{"reviews.content":{"sushi":[{"pos":160,"start":927,"end":932,"array_positions":[4]}]}},"fragments":{"reviews.content":["The staff was extremely friendly and helpful. The hotel rooms on the upper floors are much nicer than the ones on the ground level floors (1 and 2). There can be slight problems with the toilets and a…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_21727","score":0.3912784675348057,"locations":{"reviews.content":{"sushi":[{"pos":683,"start":3587,"end":3592,"array_positions":[0]}]}},"fragments":{"reviews.content":["…quite absurd. $13 for a cocktail (plus tip?). A hot dog will cost you $12 (each). I ordered some \u003cmark\u003esushi\u003c/mark\u003e and beers on the beach and rang up a $70 bill for a snack...gimme a break. Overall, a beautiful r…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_40356","score":0.36142488594737326,"locations":{"reviews.content":{"sushi":[{"pos":436,"start":2370,"end":2375,"array_positions":[7]},{"pos":444,"start":2415,"end":2420,"array_positions":[7]}]}},"fragments":{"reviews.content":["…he road (if you can ignore all of the scary people hanging around at night), and the \u003cmark\u003esushi\u003c/mark\u003e restaurant, Shinto is to die for! Best \u003cmark\u003esushi\u003c/mark\u003e, and affordable, that I have ever had! Overall we had a great st…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_1362","score":0.3512671018970044,"locations":{"reviews.content":{"sushi":[{"pos":347,"start":1749,"end":1754,"array_positions":[1]}]}},"fragments":{"reviews.content":["…Q as you watch the sunset from your balcony is pretty great! We ate out at SANSEI in Kapalua for \u003cmark\u003esushi\u003c/mark\u003e and it was quite good. This is a great condo style set up for familys or couples, and its so nice…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_21889","score":0.3444751401787315,"locations":{"reviews.content":{"sushi":[{"pos":359,"start":1956,"end":1961,"array_positions":[0]},{"pos":167,"start":899,"end":904,"array_positions":[6]}]}},"fragments":{"reviews.content":["…and those who want privacy. I saw some couple with kids, but not much. Overall, I love Ritz Carlton! The place has a \u003cmark\u003esushi\u003c/mark\u003e bar, lounge, the famous Banyan Tree restaurant, a beach restaurant, etc. YOU …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_33342","score":0.3165258739135141,"locations":{"reviews.content":{"sushi":[{"pos":176,"start":891,"end":896,"array_positions":[0]}]}},"fragments":{"reviews.content":["…ners at the hotel. The Strip house has the best salad, Blossoms was a great family style chinese/\u003cmark\u003esushi\u003c/mark\u003e meal. The italian restaurant food was fabulous ( service was a little slow). Our most favorite pl…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_15917","score":0.30752103535561687,"locations":{"reviews.content":{"sushi":[{"pos":214,"start":1085,"end":1090,"array_positions":[1]},{"pos":225,"start":1145,"end":1150,"array_positions":[1]}]}},"fragments":{"reviews.content":["…g center across the street from resort. There is also a nice small \u003cmark\u003esushi\u003c/mark\u003e bar about a block up the street that served excellent \u003cmark\u003esushi\u003c/mark\u003e and the miso soup was to die for. I visited a few other hotels in t…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_26140","score":0.27872942708496573,"locations":{"reviews.content":{"sushi":[{"pos":170,"start":944,"end":949,"array_positions":[6]}]}},"fragments":{"reviews.content":["…it was worth it. The hotel also provides a car service and is also convenient to the SF public transportation system. Restaurants close by can be touristy but I enjoyed a small \u003cmark\u003esushi\u003c/mark\u003e place on Hyde: Gr…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_25809","score":0.2689145863880359,"locations":{"reviews.content":{"sushi":[{"pos":269,"start":1408,"end":1413,"array_positions":[3]}]}},"fragments":{"reviews.content":["…nd around 10 minutes to the China Ferry Terminal (Macau). There is no restaurant on site, but there is a bar on level 1 - numerous eating places right outside including a great \u003cmark\u003esushi\u003c/mark\u003e place and a 7/11 …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_21729","score":0.2600991689671452,"locations":{"reviews.content":{"sushi":[{"pos":88,"start":450,"end":455,"array_positions":[5]},{"pos":114,"start":583,"end":588,"array_positions":[5]}]}},"fragments":{"reviews.content":["…s amazing. The next day we ate \u003cmark\u003eSushi\u003c/mark\u003e at Kincha's. Two buffets and an additional Rainbow Roll, plus a tea and glass of wine and we walked out the door for $160. The \u003cmark\u003esushi\u003c/mark\u003e, even for a buffet, was as goo…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_16630","score":0.25801422183894807,"locations":{"reviews.content":{"sushi":[{"pos":154,"start":793,"end":798,"array_positions":[0]}]}},"fragments":{"reviews.content":["…fied.\" There are many restaurants within walking distance so no need to drive around. We enjoyed \"Silk\" --they serve \u003cmark\u003esushi\u003c/mark\u003e, and chinese cuisine. Parking is a bit much at $20/day valet or $15 self-park…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_15918","score":0.24681967712010963,"locations":{"reviews.content":{"sushi":[{"pos":366,"start":1975,"end":1980,"array_positions":[7]}]}},"fragments":{"reviews.content":["…ocean close to the beach. It's included in your stay. The hotel has a few different restaurants- \u003cmark\u003esushi\u003c/mark\u003e, steak, italian, breakfast buffet. Did all of them except for the italian. Enjoyed all of them, l…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_15915","score":0.2374830627410517,"locations":{"reviews.content":{"sushi":[{"pos":530,"start":2790,"end":2795,"array_positions":[5]}]}},"fragments":{"reviews.content":["…blocks down, then a sharp right.. there is a restaurant called Tangerine/ Liquid. And a roof top \u003cmark\u003esushi\u003c/mark\u003e bar on the 17th foor, open roof that overlooks the ocean. Prices are ok. There is also local shop…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_16180","score":0.23675288624349838,"locations":{"reviews.content":{"sushi":[{"pos":64,"start":350,"end":355,"array_positions":[6]}]}},"fragments":{"reviews.content":["…nt theme everyday were good. The Japanese restaurant served a great salmon, but you will not get \u003cmark\u003esushi\u003c/mark\u003e or shimie in any quantity or quality. The other two were Caribbean and a Steak house, which were …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_16237","score":0.23097346570500746,"locations":{"reviews.content":{"sushi":[{"pos":1591,"start":8923,"end":8928,"array_positions":[1]},{"pos":1614,"start":9058,"end":9063,"array_positions":[1]}]}},"fragments":{"reviews.content":["…sented with a lovely plate of \u003cmark\u003esushi\u003c/mark\u003e and a menu. Next you are given a choice of several starters, entrees and dessert. We thoroughly enjoyed the Dominican version of \u003cmark\u003esushi\u003c/mark\u003e, then tempura, a wonderful sh…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_37318","score":0.19489984486324483,"locations":{"reviews.content":{"sushi":[{"pos":247,"start":1287,"end":1292,"array_positions":[6]}]}},"fragments":{"reviews.content":["…Our kids, who are not adventurous eaters, had to be convinced that there would be more than just \u003cmark\u003eSushi\u003c/mark\u003e at the Japanese restaurant. They were pleasantly surprised. In fact, it was their favorite restau…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_21723","score":0.17645695052787527,"locations":{"reviews.content":{"sushi":[{"pos":605,"start":3178,"end":3183,"array_positions":[8]}]}},"fragments":{"reviews.content":["…d dinner at humuhumunukunukuapuaa and it was very nice. My wife had lobster and steak, and I had \u003cmark\u003esushi\u003c/mark\u003e and fresh fish ceviche. Both were nicely prepared. They told me the fridge would be $25 for the w…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_16045","score":0.14692885515216011,"locations":{"reviews.content":{"sushi":[{"pos":731,"start":3846,"end":3851,"array_positions":[6]}]}},"fragments":{"reviews.content":["…got a reservation at the Oriental Restaurant (the entr茅e was so overcooked, it was inedible. The \u003cmark\u003esushi\u003c/mark\u003e was good.).Friday morning we ate the buffet breakfast (one side is more American and one side Eur…"]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"term":"sushi","field":"reviews.content"},"size":50,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":20,"max_score":0.5115191736068728,"took":687046,"facets":null} + +
      • + +
    • + Unanalyzed Term Query with Fuzziness Level of 2, which is almost identical to the immediately preceding query; but which this time + specifies a fuzziness factor of 2, allowing partial matches + to be made. The + output from this query can be compared to that of the one immediately preceding. + ----> {"indexName":"travel-sample-index-stored","size":50,"highlight":{"style":"html"},"query":{"field":"reviews.content","fuzziness":2,"term":"sushi"}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_9905","score":0.1447082094709048,"locations":{"reviews.content":{"bush":[{"pos":109,"start":640,"end":644,"array_positions":[3]}],"rush":[{"pos":64,"start":357,"end":361,"array_positions":[3]}]}},"fragments":{"reviews.content":["…l, it was spotlessly clean. I liked the breakfast area, although it did get a bit noisy when the \"\u003cmark\u003erush\u003c/mark\u003e\" hit about 8:00 a.m. on a Saturday morning. The room was similarly clean and comfortable. The wir…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_21727","score":0.13457791408633243,"locations":{"reviews.content":{"lush":[{"pos":68,"start":369,"end":373,"array_positions":[1]}],"sushi":[{"pos":683,"start":3587,"end":3592,"array_positions":[0]}]}},"fragments":{"reviews.content":["…quite absurd. $13 for a cocktail (plus tip?). A hot dog will cost you $12 (each). I ordered some \u003cmark\u003esushi\u003c/mark\u003e and beers on the beach and rang up a $70 bill for a snack...gimme a break. Overall, a beautiful r…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_21723","score":0.10588210962113358,"locations":{"reviews.content":{"lush":[{"pos":40,"start":199,"end":203,"array_positions":[5]}],"rush":[{"pos":252,"start":1367,"end":1371,"array_positions":[6]}],"slush":[{"pos":732,"start":3827,"end":3832,"array_positions":[8]}],"sushi":[{"pos":605,"start":3178,"end":3183,"array_positions":[8]}]}},"fragments":{"reviews.content":["…o relax. The hotel is set in beautiful grounds with water features and sculptures everywhere. The \u003cmark\u003elush\u003c/mark\u003e gardens are constantly being worked on - we thought they were beautifully kept. The hotel's facil…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_33899","score":0.10480406598230554,"locations":{"reviews.content":{"lush":[{"pos":105,"start":584,"end":588,"array_positions":[2]},{"pos":13,"start":84,"end":88,"array_positions":[4]}],"rush":[{"pos":228,"start":1288,"end":1292,"array_positions":[4]}]}},"fragments":{"reviews.content":["…elf was magnificent and very clean. The landscaping was extremely well taken care of, it was very \u003cmark\u003elush\u003c/mark\u003e and tropical. The facilities were very good and pretty much offered anything that you could imagi…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_26211","score":0.07063424993300373,"locations":{"reviews.content":{"push":[{"pos":114,"start":603,"end":607,"array_positions":[6]}],"sisli":[{"pos":20,"start":102,"end":107,"array_positions":[1]}],"xishi":[{"pos":33,"start":188,"end":193,"array_positions":[3]}]}},"fragments":{"reviews.content":["…e at a nearby hotel. for four nights at the Mamara \u003cmark\u003eSisli\u003c/mark\u003e hotel. The hotel is not in the main tourist area, and it is quite expensive to get there by cab (I didn't have adequate Turkish or stamina to t…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_12924","score":0.06130418173247418,"locations":{"reviews.content":{"mushy":[{"pos":108,"start":554,"end":559,"array_positions":[3]}],"sasha":[{"pos":132,"start":703,"end":708,"array_positions":[4]}]}},"fragments":{"reviews.content":["… and it got loud. We could also hear people on the street so this may not be a big problem now, it could a problem for some at peak season. The bed was \u003cmark\u003emushy\u003c/mark\u003e, thankfully I didn't wake up with a back a…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_21729","score":0.05952962901832172,"locations":{"reviews.content":{"rush":[{"pos":293,"start":1506,"end":1510,"array_positions":[7]}],"sushi":[{"pos":88,"start":450,"end":455,"array_positions":[5]},{"pos":114,"start":583,"end":588,"array_positions":[5]}]}},"fragments":{"reviews.content":["…s amazing. The next day we ate \u003cmark\u003eSushi\u003c/mark\u003e at Kincha's. Two buffets and an additional Rainbow Roll, plus a tea and glass of wine and we walked out the door for $160. The \u003cmark\u003esushi\u003c/mark\u003e, even for a buffet, was as goo…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_21845","score":0.05666025238268102,"locations":{"reviews.content":{"lush":[{"pos":7,"start":37,"end":41,"array_positions":[1]}]}},"fragments":{"reviews.content":["…kena. Property is \u003cmark\u003elush\u003c/mark\u003e, beach is nice. Room was spacious. very reasonably priced compared to wailea. nice breakfast buffet. Nice touches at the pool. Servers with water and frozen grapes. Food at pool…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_32172","score":0.05162526868803317,"locations":{"reviews.content":{"lush":[{"pos":78,"start":398,"end":402,"array_positions":[0]}]}},"fragments":{"reviews.content":["… ever stay, coming almost every year when we can. We love the proximity to Main St. and all the restuarants and shopping. We absoutly love the \u003cmark\u003elush\u003c/mark\u003e and beautiful gardens the pools and piece and Quite.…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_21725","score":0.04196935945680559,"locations":{"reviews.content":{"pushy":[{"pos":89,"start":485,"end":490,"array_positions":[0]}],"rush":[{"pos":843,"start":4418,"end":4422,"array_positions":[0]}]}},"fragments":{"reviews.content":["…lax there, not worry about racing for good seats, there are not 1000's of screaming children and \u003cmark\u003epushy\u003c/mark\u003e adults. I just dont know how else to describe it other than \"down market,\" which is not how a hot…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_3623","score":0.03880039965703977,"locations":{"reviews.content":{"push":[{"pos":280,"start":1522,"end":1526,"array_positions":[0]}]}},"fragments":{"reviews.content":["…woman behind the front desk left her spot as I walked in the lobby and beat me to the elevator to \u003cmark\u003epush\u003c/mark\u003e the buttons for me (both outside the elevator and in), I was impressed; I soon came to expect tha…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_25798","score":0.03798801962666054,"locations":{"reviews.content":{"push":[{"pos":366,"start":2028,"end":2032,"array_positions":[2]},{"pos":502,"start":2719,"end":2723,"array_positions":[2]}]}},"fragments":{"reviews.content":["…ll, a very attractive room, with automatic curtains so you can just lie in bed in the morning and \u003cmark\u003epush\u003c/mark\u003e a button to see the view of the canal (and people breakfasting at the hotel opposite). Our daught…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_16436","score":0.037757007718035085,"locations":{"reviews.content":{"lush":[{"pos":69,"start":390,"end":394,"array_positions":[3]}],"rush":[{"pos":401,"start":2071,"end":2075,"array_positions":[7]}]}},"fragments":{"reviews.content":["…r opinion of the resort and the (non-existing) customer service:Grounds: Beautiful landscape with \u003cmark\u003elush\u003c/mark\u003e greenery, lagoons, fish, flamingoes and, among other wild life, a bunch of geese. One of them, a …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_12242","score":0.03746367785606021,"locations":{"reviews.content":{"rush":[{"pos":58,"start":306,"end":310,"array_positions":[0]}]}},"fragments":{"reviews.content":["…the Hospital. The Interstate is right there and only a 15 minute drive to downtown, except during \u003cmark\u003erush\u003c/mark\u003e hour. The staff was friendly and helpful (save for one of the women who took care of the breakfas…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_7387","score":0.03727408874671707,"locations":{"reviews.content":{"rush":[{"pos":683,"start":3645,"end":3649,"array_positions":[1]}]}},"fragments":{"reviews.content":["…ch nicer/cleaner. We were running late on our way back to the airport (and it was at the start of \u003cmark\u003erush\u003c/mark\u003e hour), but he knew the shortcuts and still got us to the airport in just over 30 minutes. Also: O…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_21844","score":0.03565168401900037,"locations":{"reviews.content":{"sushi":[{"pos":160,"start":927,"end":932,"array_positions":[4]}]}},"fragments":{"reviews.content":["The staff was extremely friendly and helpful. The hotel rooms on the upper floors are much nicer than the ones on the ground level floors (1 and 2). There can be slight problems with the toilets and a…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_21852","score":0.03447494168235994,"locations":{"reviews.content":{"lush":[{"pos":143,"start":754,"end":758,"array_positions":[0]},{"pos":97,"start":538,"end":542,"array_positions":[4]}]}},"fragments":{"reviews.content":["…t well for us though. We got on the top floor \u0026 had a garden view, but when your view overlooks a \u003cmark\u003elush\u003c/mark\u003e golf course \u0026 a mountain, it's still not bad, so I didnt miss that we couldnt really see the ocea…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_16527","score":0.0328110214969065,"locations":{"reviews.content":{"sushi":[{"pos":168,"start":898,"end":903,"array_positions":[5]},{"pos":193,"start":1040,"end":1045,"array_positions":[5]}]}},"fragments":{"reviews.content":["…aff were superb. Sancho's \u003cmark\u003eSushi\u003c/mark\u003e Bar was the only disappointment - food was not good and prices were too high. Coming from San Francisco and Washington DC we have great \u003cmark\u003esushi\u003c/mark\u003e restaurants. Sancho's coul…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_25596","score":0.032447069719011096,"locations":{"reviews.content":{"susan":[{"pos":28,"start":174,"end":179,"array_positions":[1]}]}},"fragments":{"reviews.content":["…r Union Square, amazing value and super friendly and helpful staff. Special mention to \u003cmark\u003eSusan\u003c/mark\u003e who helped us with last minute city trip and airport transfer. The place has a really charming ambience and…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_23198","score":0.029378949585684384,"locations":{"reviews.content":{"sushi":[{"pos":209,"start":1121,"end":1126,"array_positions":[0]},{"pos":215,"start":1150,"end":1155,"array_positions":[0]}]}},"fragments":{"reviews.content":["… Encore/Wynn. Starbucks for breakfast is half the price of the cafe in the hotel. Also, \u003cmark\u003eSushi\u003c/mark\u003e Ra has good half price \u003cmark\u003esushi\u003c/mark\u003e from 5 - 7pm. (4) Boarding pass: Encore/Wynn lets you print it for free in th…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_32861","score":0.029224826417803666,"locations":{"reviews.content":{"lush":[{"pos":14,"start":79,"end":83,"array_positions":[0]}]}},"fragments":{"reviews.content":["Loved it and everything about it! Am planning to return soon! Amazing service, \u003cmark\u003elush\u003c/mark\u003e rooms with views to die for! Beautiful poor area and great bar staff."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_3611","score":0.028944736969211397,"locations":{"reviews.content":{"rush":[{"pos":99,"start":538,"end":542,"array_positions":[0]}]}},"fragments":{"reviews.content":["…l compliment. The offer of a glass of wine when I first entered the lobby after stressing through \u003cmark\u003erush\u003c/mark\u003e hour traffic was very welcome. I know there have been lots of complaints about noise but we had a…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_12238","score":0.028765091379839108,"locations":{"reviews.content":{"susan":[{"pos":30,"start":147,"end":152,"array_positions":[0]},{"pos":73,"start":375,"end":380,"array_positions":[0]}]}},"fragments":{"reviews.content":["…all the way to US 192 to find a hotel. I spoke with a lovely woman named \u003cmark\u003eSusan\u003c/mark\u003e to see it they had vacancies and we were pleased to hear that they did. Upon arrival, we stopped at a guardhouse with sec…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_12923","score":0.02759553852192764,"locations":{"reviews.content":{"rush":[{"pos":167,"start":969,"end":973,"array_positions":[6]}]}},"fragments":{"reviews.content":["…with 2 lattes was $50. While opting for convience we were actually delayed, gobbled our food in a \u003cmark\u003erush\u003c/mark\u003e, and made late to our appointment with Far Niente. Overall, great hotel, but avoid the restaurant…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_25670","score":0.025817927796274585,"locations":{"reviews.content":{"hush":[{"pos":57,"start":278,"end":282,"array_positions":[7]}]}},"fragments":{"reviews.content":["…the only hint of what a great place this is. Our room moved us from the hustle of downtown to the \u003cmark\u003ehush\u003c/mark\u003e of a sweet large room set up like a charming home, with a fireplace, french doors over a terrace,…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_25801","score":0.025799683875495236,"locations":{"reviews.content":{"lush":[{"pos":106,"start":549,"end":553,"array_positions":[1]}]}},"fragments":{"reviews.content":["…ll they have put a lot of thought and pride in the furnishings and decor - everything was new and \u003cmark\u003elush\u003c/mark\u003e. We stayed on the top floor and one of the bedrooms was in the turret, which was quite special. Y…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_2483","score":0.025714695663631146,"locations":{"reviews.content":{"pushy":[{"pos":97,"start":543,"end":548,"array_positions":[2]}]}},"fragments":{"reviews.content":["… although I was aware of their cancellation policy (I should note that I was rather impolite and \u003cmark\u003epushy\u003c/mark\u003e, but they put up with me and were very patient). Their staff recited their policy as I had expect…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_21649","score":0.024997569779925535,"locations":{"reviews.content":{"push":[{"pos":731,"start":3810,"end":3814,"array_positions":[0]}]}},"fragments":{"reviews.content":["…er's hand. She was still in the elevator and I had to put my body in the door of the elevator and \u003cmark\u003epush\u003c/mark\u003e it open. My daughter was screaming and I was completely panicked. The same thing happened to my h…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_38525","score":0.024583837464192605,"locations":{"reviews.content":{"lush":[{"pos":324,"start":1717,"end":1721,"array_positions":[1]}]}},"fragments":{"reviews.content":["…e beach, but we were happy with the set-up of the pool areas. As for the complaints about lack of \u003cmark\u003elush\u003c/mark\u003e vegetation, again-I feel that the people complaining just don't understand the design. There was …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_14005","score":0.024435929745638147,"locations":{"reviews.content":{"rush":[{"pos":321,"start":1763,"end":1767,"array_positions":[0]}]}},"fragments":{"reviews.content":["…taurant, and it was really worth it. The food was great and the athmosphere quite nice. We had to \u003cmark\u003erush\u003c/mark\u003e out though, since the children had heard about \"smors\" session at the \"beach\" and it was about to…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_1362","score":0.02421679276309202,"locations":{"reviews.content":{"sushi":[{"pos":347,"start":1749,"end":1754,"array_positions":[1]}]}},"fragments":{"reviews.content":["…Q as you watch the sunset from your balcony is pretty great! We ate out at SANSEI in Kapalua for \u003cmark\u003esushi\u003c/mark\u003e and it was quite good. This is a great condo style set up for familys or couples, and its so nice…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_16630","score":0.02279911103661969,"locations":{"reviews.content":{"sushi":[{"pos":154,"start":793,"end":798,"array_positions":[0]}]}},"fragments":{"reviews.content":["…fied.\" There are many restaurants within walking distance so no need to drive around. We enjoyed \"Silk\" --they serve \u003cmark\u003esushi\u003c/mark\u003e, and chinese cuisine. Parking is a bit much at $20/day valet or $15 self-park…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_26236","score":0.0226672810151682,"locations":{"reviews.content":{"push":[{"pos":250,"start":1382,"end":1386,"array_positions":[2]}]}},"fragments":{"reviews.content":["…you have to travel by taxi which ranges from 20 AEDs to 60+ AEDs. The hotel security staff try to \u003cmark\u003epush\u003c/mark\u003e private taxis and city tours which we found were were at least 300% costlier when we compared sim…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_26219","score":0.022029313120907404,"locations":{"reviews.content":{"lush":[{"pos":162,"start":822,"end":826,"array_positions":[2]}]}},"fragments":{"reviews.content":["…ut or you want to keep up your skills during a summer vacation. The grass on the soccer field was \u003cmark\u003elush\u003c/mark\u003e it felt like you were walking on a carpet, no holes and nice and level. A+ And for those of you w…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_23634","score":0.021739714258808288,"locations":{"reviews.content":{"push":[{"pos":209,"start":1057,"end":1061,"array_positions":[2]}]}},"fragments":{"reviews.content":["…that lifts the gate handle up. This happened three times that the gates would not open. We had to \u003cmark\u003epush\u003c/mark\u003e the assistance buzzer. The third time it happened my husband pushed the assistance buzzer but no …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_25667","score":0.021530662926702065,"locations":{"reviews.content":{"hush":[{"pos":169,"start":899,"end":903,"array_positions":[5]}]}},"fragments":{"reviews.content":["…counts to most anything you want to see. Also, if you want to see a show, go to the TIX stand (right accross from the hotel basically) for 1/2-price same-day shows. We saw \"\u003cmark\u003eHush\u003c/mark\u003e Up...Sweet Charolette\"…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_30018","score":0.021368650206674448,"locations":{"reviews.content":{"lush":[{"pos":75,"start":426,"end":430,"array_positions":[2]}]}},"fragments":{"reviews.content":["…ildings and destinations. The drinks were a little expensive, but it's a resort and expected. Hard floors were perfect for getting sand in the room and \u003cmark\u003elush\u003c/mark\u003e plants were perfect in creating the island …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_16458","score":0.020418097547174285,"locations":{"reviews.content":{"gushy":[{"pos":90,"start":544,"end":549,"array_positions":[4]}]}},"fragments":{"reviews.content":["…tart with reception: nice and discrete ground floor welcome, warm and enthusiastic without being \u003cmark\u003egushy\u003c/mark\u003e from an impeccably groomed and professional team. It also smelled gorgeous. We were terribly earl…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_2484","score":0.02017983949060119,"locations":{"reviews.content":{"push":[{"pos":20,"start":110,"end":114,"array_positions":[7]}]}},"fragments":{"reviews.content":["… it's only attached to Terminal 3 so you might need to \u003cmark\u003epush\u003c/mark\u003e bags and trolleys around for a while if you arrive at any other terminal - not the hotel's fault. Guests are mostly passengers in transit an…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_12240","score":0.02015225258532386,"locations":{"reviews.content":{"push":[{"pos":25,"start":120,"end":124,"array_positions":[2]}]}},"fragments":{"reviews.content":["… just incompetent. If your not hiring a car DO NOT let them \u003cmark\u003epush\u003c/mark\u003e Ceasers Transport on you. They're paid comission by this unreliable, over priced taxi firm. Insted look up a local taxi, which we found…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_8962","score":0.019768082806908838,"locations":{"reviews.content":{"lush":[{"pos":47,"start":250,"end":254,"array_positions":[4]}]}},"fragments":{"reviews.content":["…, Sea World, Old Town San Diego (great restaurants in Old Town), parks etc. The resort feels very \u003cmark\u003elush\u003c/mark\u003e with many pools to choose from, and the fire pits are a nice touch. Staff was helpful with direct…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_20738","score":0.018655401016647234,"locations":{"reviews.content":{"push":[{"pos":182,"start":1001,"end":1005,"array_positions":[4]}]}},"fragments":{"reviews.content":["…to the details of many of the offers and asks for proof from the guest. I'm sure they get lots of \u003cmark\u003epush\u003c/mark\u003e back considering the level of clientele staying in this hotel, but in my case since I had printed…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_15912","score":0.018424948398114756,"locations":{"reviews.content":{"pushy":[{"pos":310,"start":1603,"end":1608,"array_positions":[0]}]}},"fragments":{"reviews.content":["…nd loved having the option. There were a few tables of vendors selling jewelry and trinkets, not \u003cmark\u003epushy\u003c/mark\u003e at all and in fact I had to work my way thru the crowd of people around the tables to take a look…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_21889","score":0.018168262094237917,"locations":{"reviews.content":{"sushi":[{"pos":359,"start":1956,"end":1961,"array_positions":[0]},{"pos":167,"start":899,"end":904,"array_positions":[6]}]}},"fragments":{"reviews.content":["…and those who want privacy. I saw some couple with kids, but not much. Overall, I love Ritz Carlton! The place has a \u003cmark\u003esushi\u003c/mark\u003e bar, lounge, the famous Banyan Tree restaurant, a beach restaurant, etc. YOU …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_33896","score":0.0178679897843285,"locations":{"reviews.content":{"pushy":[{"pos":172,"start":890,"end":895,"array_positions":[8]}],"slush":[{"pos":352,"start":1915,"end":1920,"array_positions":[2]}]}},"fragments":{"reviews.content":["…y good.The drinks would be better if they were made with fresh fruit, instead of slushee machine \u003cmark\u003eslush\u003c/mark\u003e, however they were decent.The beds on the beach were amazing, but ignore the rules and reserve on…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_4398","score":0.017754407511884743,"locations":{"reviews.content":{"lush":[{"pos":17,"start":92,"end":96,"array_positions":[6]}]}},"fragments":{"reviews.content":["…it blew my expectations away. The grounds are \u003cmark\u003elush\u003c/mark\u003e and simply amazing so take a nice long walk. The country breakfast was a real treat, actually a huge spread. We had a couples massage in The Redwood …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_25809","score":0.017249328529746855,"locations":{"reviews.content":{"sushi":[{"pos":269,"start":1408,"end":1413,"array_positions":[3]}]}},"fragments":{"reviews.content":["…nd around 10 minutes to the China Ferry Terminal (Macau). There is no restaurant on site, but there is a bar on level 1 - numerous eating places right outside including a great \u003cmark\u003esushi\u003c/mark\u003e place and a 7/11 …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_40093","score":0.017191674819673815,"locations":{"reviews.content":{"rush":[{"pos":490,"start":2695,"end":2699,"array_positions":[2]}]}},"fragments":{"reviews.content":["… had an insanely early checkout time (five-ish in the a.m.), and Lindsay realized that I was in a \u003cmark\u003erush\u003c/mark\u003e to get to O'Hare/ORD for my morning cross-border flight back north. He noticed that I was getting…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_33894","score":0.01682731827292802,"locations":{"reviews.content":{"lush":[{"pos":621,"start":3407,"end":3411,"array_positions":[1]},{"pos":61,"start":319,"end":323,"array_positions":[5]}]}},"fragments":{"reviews.content":["…eserves a medal. the housekeeping staff and groundskeepers are wonderful and the surroundings are \u003cmark\u003elush\u003c/mark\u003e and beautiful.We really wanted to love this place but couldn't. The beautiful grounds couldn't ma…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_33342","score":0.016694165604765138,"locations":{"reviews.content":{"sushi":[{"pos":176,"start":891,"end":896,"array_positions":[0]}]}},"fragments":{"reviews.content":["…ners at the hotel. The Strip house has the best salad, Blossoms was a great family style chinese/\u003cmark\u003esushi\u003c/mark\u003e meal. The italian restaurant food was fabulous ( service was a little slow). Our most favorite pl…"]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"term":"sushi","prefix_length":0,"fuzziness":2,"field":"reviews.content"},"size":50,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":99,"max_score":0.1447082094709048,"took":8071632,"facets":null} + +
      • + +
    • + Match Phrase Query, using Analysis, for searching on a phrase. + ----> {"indexName":"travel-sample-index-stored","size":10,"highlight":{"style":"html"},"query":{"field":"description","match_phrase":"Eiffel Tower"}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_21726","score":2.6355418515879965,"locations":{"description":{"eiffel":[{"pos":19,"start":107,"end":113,"array_positions":null}],"tower":[{"pos":20,"start":114,"end":119,"array_positions":null}]}},"fragments":{"description":["The executive and deluxe rooms offer a breathtaking views of the Louvre, the Jardin des Tuileries, and the \u003cmark\u003eEiffel\u003c/mark\u003e \u003cmark\u003eTower\u003c/mark\u003e. Classic Parisian-style hotel next to shopping and cultural hot spots."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_21652","score":2.3882543011509223,"locations":{"description":{"eiffel":[{"pos":11,"start":53,"end":59,"array_positions":null}],"tower":[{"pos":12,"start":60,"end":65,"array_positions":null}]}},"fragments":{"description":["…e hotel very close to the \u003cmark\u003eEiffel\u003c/mark\u003e \u003cmark\u003eTower\u003c/mark\u003e, in a nice, quiet and green area of Grenelle. Despite the closeness, only about a quarter of the rooms, designated as such and commanding higher rates, feature a…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_21675","score":2.0541928282431576,"locations":{"description":{"eiffel":[{"pos":30,"start":164,"end":170,"array_positions":null}],"tower":[{"pos":31,"start":171,"end":176,"array_positions":null}]}},"fragments":{"description":["…djacent to the Palais du Congres. It enjoys an unobstructed line of sight towards the \u003cmark\u003eEiffel\u003c/mark\u003e \u003cmark\u003eTower\u003c/mark\u003e from the rooms facing it and its top-floor bar. After Hyatt took over the property, it is being gradu…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_21657","score":1.9430718762111634,"locations":{"description":{"eiffel":[{"pos":33,"start":192,"end":198,"array_positions":null}],"tower":[{"pos":34,"start":199,"end":204,"array_positions":null}]}},"fragments":{"description":["…and underwent extensive renovation to its rooms and public spaces. It continues to offer close-up views of the \u003cmark\u003eEiffel\u003c/mark\u003e \u003cmark\u003eTower\u003c/mark\u003e from the balconies of its relatively spacious for Paris Deluxe (32 sqm) and …"]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"match_phrase":"Eiffel Tower","field":"description"},"size":10,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":4,"max_score":2.6355418515879965,"took":378475,"facets":null} + +
      • + +
    • + Phrase Query, without Analysis, for searching on a phrase without analysis supported. + ----> {"indexName":"travel-sample-index-stored","size":10,"highlight":{"style":"html"},"query":{"field":"description","terms":["dorm","rooms"]}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_6351","score":3.66056027016252,"locations":{"description":{"dorm":[{"pos":5,"start":26,"end":30,"array_positions":null}],"rooms":[{"pos":6,"start":31,"end":36,"array_positions":null}]}},"fragments":{"description":["Ensuite private rooms and \u003cmark\u003edorm\u003c/mark\u003e \u003cmark\u003erooms\u003c/mark\u003e. Reception open from 0800-1030 and 1500-2230."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_20956","score":3.6086540532893583,"locations":{"description":{"dorm":[{"pos":3,"start":14,"end":18,"array_positions":null}],"rooms":[{"pos":4,"start":19,"end":24,"array_positions":null}]}},"fragments":{"description":["small ensuite \u003cmark\u003edorm\u003c/mark\u003e \u003cmark\u003erooms\u003c/mark\u003e and private guest rooms, sea-front location."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_25324","score":3.363422281385767,"locations":{"description":{"dorm":[{"pos":7,"start":34,"end":38,"array_positions":null}],"rooms":[{"pos":8,"start":39,"end":44,"array_positions":null}]}},"fragments":{"description":["A friendly and clean hostel, with \u003cmark\u003edorm\u003c/mark\u003e \u003cmark\u003erooms\u003c/mark\u003e at $17-19/night and private rooms in the $40 range."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_22460","score":2.927333486354325,"locations":{"description":{"dorm":[{"pos":4,"start":17,"end":21,"array_positions":null}],"rooms":[{"pos":5,"start":22,"end":27,"array_positions":null}]}},"fragments":{"description":["Twin, double and \u003cmark\u003edorm\u003c/mark\u003e \u003cmark\u003erooms\u003c/mark\u003e are available plus a wonderful sunny lounge, overlooking the main street."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_25152","score":2.7462228531301713,"locations":{"description":{"dorm":[{"pos":21,"start":101,"end":105,"array_positions":null}],"rooms":[{"pos":22,"start":106,"end":111,"array_positions":null}]}},"fragments":{"description":["Hostel with easy access to the historic Gaslamp District. They have 4-bed and 6-bed female and co-ed \u003cmark\u003edorm\u003c/mark\u003e \u003cmark\u003erooms\u003c/mark\u003e as well as some private rooms."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_24606","score":2.0606939023227353,"locations":{"description":{"dorm":[{"pos":26,"start":144,"end":148,"array_positions":null}],"rooms":[{"pos":27,"start":149,"end":154,"array_positions":null}]}},"fragments":{"description":["…nside Kinloch Castle, but due to the deteriorating condition, it has moved into an adjacent temporary building. It has 8 \u003cmark\u003edorm\u003c/mark\u003e \u003cmark\u003erooms\u003c/mark\u003e, each with 4 beds. All accommodation is self-catering, with a kitche…"]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"terms":["dorm","rooms"],"field":"description"},"size":10,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":6,"max_score":3.66056027016252,"took":183167,"facets":null} + +
      • + +
    • + Query String Query, showing how a query string is specified as search-input. + ----> {"indexName":"travel-sample-index-unstored","size":10,"query":{"query":"description: Imperial"}} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"hotel_26223","score":1.0839661760870005,"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"query":"description: Imperial"},"size":10,"from":0,"highlight":null,"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":1,"max_score":1.0839661760870005,"took":121026,"facets":null} + +
      • + +
    • + Conjunction Query, whereby two separate queries are defined and then run as part of the search, + with only the matches returned by both included in the result-object. + ----> {"indexName":"travel-sample-index-stored","size":10,"highlight":{"style":"html"},"query":{"conjuncts":[{"field":"reviews.content","match":"La Rue Saint Denis!!"},{"field":"description","match":"boutique"}]}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_26223","score":1.3959054545023135,"locations":{"description":{"boutique":[{"pos":6,"start":26,"end":34,"array_positions":null}]},"reviews.content":{"denis":[{"pos":88,"start":443,"end":448,"array_positions":[0]}],"la":[{"pos":85,"start":430,"end":432,"array_positions":[0]}],"rue":[{"pos":86,"start":433,"end":436,"array_positions":[0]}],"saint":[{"pos":87,"start":437,"end":442,"array_positions":[0]}]}},"fragments":{"description":["…hotel with a \u003cmark\u003eboutique\u003c/mark\u003e feel and very large bathrooms. Rooms are equipped with high-speed Internet, stereos and large flat panel televisions, and you can ask for the few rooms with Japanese style amenit…"],"reviews.content":["…libre had thieves? After all, this is supposed to be a 4 stars hotel, not a bedsit next to \u003cmark\u003eLa\u003c/mark\u003e \u003cmark\u003eRue\u003c/mark\u003e \u003cmark\u003eSaint\u003c/mark\u003e \u003cmark\u003eDenis\u003c/mark\u003e!!! I complained to the manager, Ms. Laura Lamblin, who told me that I had to go to the pol…"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_16349","score":0.8796689449270705,"locations":{"description":{"boutique":[{"pos":3,"start":10,"end":18,"array_positions":null}]},"reviews.content":{"la":[{"pos":124,"start":654,"end":656,"array_positions":[1]},{"pos":3,"start":12,"end":14,"array_positions":[3]}]}},"fragments":{"description":["Four star \u003cmark\u003eboutique\u003c/mark\u003e hotel popular with business travelers with its sleek and contemporary design and furnishings."],"reviews.content":["…s, etc. etc. etc.The food was very good for the most part. There were plenty of choices. The two a-\u003cmark\u003ela\u003c/mark\u003e-carte restaurants were average. I found the food better at the buffet restaurant in the hotel.The …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_25245","score":0.7530974096057772,"locations":{"description":{"boutique":[{"pos":10,"start":48,"end":56,"array_positions":null}]},"reviews.content":{"la":[{"pos":78,"start":449,"end":451,"array_positions":[4]}]}},"fragments":{"description":["Right on the Pacific Ocean and features a retro \u003cmark\u003eboutique\u003c/mark\u003e-style décor, a popular onsite restaurant, and private guestroom balconies."],"reviews.content":["…quired, presentation was beautiful, taste was excellent and variety was always a pleasant surprise! The staff always went above and beyond to see to our every request. Linda Alumbaugh, Baton Rouge, \u003cmark\u003eLA\u003c/mark\u003e"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_25166","score":0.6894979469980613,"locations":{"description":{"boutique":[{"pos":11,"start":54,"end":62,"array_positions":null}]},"reviews.content":{"la":[{"pos":76,"start":373,"end":375,"array_positions":[2]},{"pos":101,"start":509,"end":511,"array_positions":[2]}]}},"fragments":{"description":["Formerly the Se Sean Diego, now a Kimpton Hotel. This \u003cmark\u003eboutique\u003c/mark\u003e hotel is near Horton Square and a few blocks from the Gaslamp District. Rates start at $159 a night."],"reviews.content":["…om the beach and you can go to \u003cmark\u003eLa\u003c/mark\u003e Sagrada Familia even walking, the metro is right behind the hotel. We discovered the area and it is beatiful, specially if you go to \u003cmark\u003ela\u003c/mark\u003e Rambla del Poblenou with amazi…"]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"conjuncts":[{"match":"La Rue Saint Denis!!","field":"reviews.content","prefix_length":0,"fuzziness":0,"operator":"or"},{"match":"boutique","field":"description","prefix_length":0,"fuzziness":0,"operator":"or"}]},"size":10,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":4,"max_score":1.3959054545023135,"took":285987,"facets":null} + +
      • + + +
    • + Wild Card Query, whereby a wildcard is used in the string submitted for the search. + ----> {"indexName":"travel-sample-index-stored","size":10,"highlight":{"style":"html"},"query":{"field":"description","wildcard":"bouti*ue"}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_21663","score":3.584615143683038,"locations":{"description":{"boutique":[{"pos":3,"start":7,"end":15,"array_positions":null}]}},"fragments":{"description":["3-star \u003cmark\u003eboutique\u003c/mark\u003e hotel."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_635","score":3.2510360559760345,"locations":{"description":{"boutique":[{"pos":5,"start":18,"end":26,"array_positions":null}]}},"fragments":{"description":["4 stat town house \u003cmark\u003eboutique\u003c/mark\u003e hotel."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_13aa53f3","id":"hotel_16686","score":3.206177215390049,"locations":{"description":{"boutique":[{"pos":4,"start":14,"end":22,"array_positions":null}]}},"fragments":{"description":["4 star luxury \u003cmark\u003eboutique\u003c/mark\u003e hotel."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_16448","score":2.694909872488883,"locations":{"description":{"boutique":[{"pos":1,"start":0,"end":8,"array_positions":null}]}},"fragments":{"description":["\u003cmark\u003eBoutique\u003c/mark\u003e townhouse rooms and accommodation just off Kensington High Street."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_1501","score":2.694035322713864,"locations":{"description":{"boutique":[{"pos":2,"start":5,"end":13,"array_positions":null}]}},"fragments":{"description":["This \u003cmark\u003eboutique\u003c/mark\u003e hotel offers five unique food and beverage venues."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_25599","score":2.610662777382389,"locations":{"description":{"boutique":[{"pos":2,"start":2,"end":10,"array_positions":null}]}},"fragments":{"description":["A \u003cmark\u003eboutique\u003c/mark\u003e hotel, favored by musicians of all stripes, with free parking and breakfast."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_16522","score":2.5650913444173966,"locations":{"description":{"boutique":[{"pos":5,"start":21,"end":29,"array_positions":null}]}},"fragments":{"description":["A charming four star \u003cmark\u003eboutique\u003c/mark\u003e hotel near Victoria station."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_21849","score":2.5407855132362815,"locations":{"description":{"boutique":[{"pos":3,"start":10,"end":18,"array_positions":null}]}},"fragments":{"description":["Five-star \u003cmark\u003eboutique\u003c/mark\u003e hotel with 44 individually decorated rooms and suites."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_15918","score":2.418391425463296,"locations":{"description":{"boutique":[{"pos":1,"start":0,"end":8,"array_positions":null}]}},"fragments":{"description":["\u003cmark\u003eBoutique\u003c/mark\u003e hotel in the City Centre; Marco Pierre White restaurant on-site."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_6ddbfb54","id":"hotel_21847","score":2.2942874696887805,"locations":{"description":{"boutique":[{"pos":4,"start":19,"end":27,"array_positions":null}]}},"fragments":{"description":["Charming five-star \u003cmark\u003eboutique\u003c/mark\u003e hotel with a secluded patio and lush vertical garden."]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"wildcard":"bouti*ue","field":"description"},"size":10,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":43,"max_score":3.584615143683038,"took":2384276,"facets":null} + +
      • + +
    • + Numeric Range Query, whereby minimum and maximum numbers are specified, and matches + within the range returned. + ----> {"indexName":"travel-sample-index-unstored","size":10,"query":{"field":"id","max":10200.0,"min":10100.0}} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_13aa53f3","id":"route_10126","score":0.20746380094270872,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_13aa53f3","id":"hotel_10138","score":0.20746380094270872,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_13aa53f3","id":"landmark_10128","score":0.20746380094270872,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_13aa53f3","id":"landmark_10127","score":0.20746380094270872,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_13aa53f3","id":"landmark_10134","score":0.20746380094270872,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_13aa53f3","id":"landmark_10116","score":0.20746380094270872,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"hotel_10158","score":0.18678699332733317,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"route_10150","score":0.18678699332733317,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"route_10121","score":0.15745693430772972,"sort":["_score"]} +{"index":"travel-sample-index-unstored_3c01f77ee05090a7_aa574717","id":"route_10132","score":0.15745693430772972,"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"min":10100,"max":10200,"field":"id"},"size":10,"from":0,"highlight":null,"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":88,"max_score":0.20746380094270872,"took":3926276,"facets":null} + +
      • + +
    • + Regexp Query, whereby a regular expression is submitted, to generate the conditions for + successful matches. +----> {"indexName":"travel-sample-index-stored","size":10,"highlight":{"style":"html"},"query":{"field":"description","regexp":"[a-z]"}} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_41237","score":2.9424485879934785,"locations":{"description":{"b":[{"pos":3,"start":15,"end":16,"array_positions":null},{"pos":4,"start":17,"end":18,"array_positions":null}],"y":[{"pos":20,"start":104,"end":105,"array_positions":null}]}},"fragments":{"description":["Self contained \u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e apartment for 2 people sharing. Self catering Holiday Farmhouse to sleep 4 or 8. Pen \u003cmark\u003ey\u003c/mark\u003e Graig is 500m from Church Bay beach and the Anglesey Coastal Footpath"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_27626","score":1.5510963030486056,"locations":{"description":{"b":[{"pos":4,"start":20,"end":21,"array_positions":null},{"pos":5,"start":22,"end":23,"array_positions":null}],"y":[{"pos":8,"start":34,"end":35,"array_positions":null}]}},"fragments":{"description":["A reasonably priced \u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e near Coed-\u003cmark\u003ey\u003c/mark\u003e-Brenin with four ensuite rooms which can be either double or twin rooms. Beautiful Farm 2008, Regional Winner."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_2241","score":1.3555882835203943,"locations":{"description":{"b":[{"pos":4,"start":20,"end":21,"array_positions":null},{"pos":5,"start":22,"end":23,"array_positions":null},{"pos":32,"start":192,"end":193,"array_positions":null},{"pos":33,"start":194,"end":195,"array_positions":null}],"y":[{"pos":60,"start":352,"end":353,"array_positions":null}]}},"fragments":{"description":["…atchers are welcomed. This \u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e is conveniently located for visiting Bangor University staff and is located 10 minutes away from the soon to be opened 'Pontio' arts complex in Bangor and '\u003cmark\u003eY\u003c/mark\u003e Galeri' in …"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_26139","score":0.8486112340942231,"locations":{"description":{"b":[{"pos":3,"start":23,"end":24,"array_positions":null},{"pos":4,"start":25,"end":26,"array_positions":null}]}},"fragments":{"description":["Reasonably inexpensive \u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e lodging."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_18572d87","id":"hotel_24537","score":0.8453215360863323,"locations":{"description":{"b":[{"pos":1,"start":0,"end":1,"array_positions":null},{"pos":2,"start":2,"end":3,"array_positions":null}]}},"fragments":{"description":["\u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e 8 miles from the city center,within easy access from the A-28-A29-A151 motorways, this is an elegant Napoleon III style building in a fully secured walled park of 4 ha."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_54820232","id":"hotel_35668","score":0.7172073919299199,"locations":{"description":{"b":[{"pos":1,"start":0,"end":1,"array_positions":null},{"pos":2,"start":2,"end":3,"array_positions":null}]}},"fragments":{"description":["\u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e accommodation, 600 yards from Pennine Way."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_21036","score":0.6150102049883732,"locations":{"description":{"b":[{"pos":1,"start":0,"end":1,"array_positions":null},{"pos":2,"start":2,"end":3,"array_positions":null}]}},"fragments":{"description":["\u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e; Self-catering accommodation"]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_f4e0a48a","id":"hotel_37340","score":0.5614249644336504,"locations":{"description":{"b":[{"pos":3,"start":11,"end":12,"array_positions":null},{"pos":4,"start":13,"end":14,"array_positions":null}]}},"fragments":{"description":["Family run \u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e and licensed bar."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_15976","score":0.5099272696653835,"locations":{"description":{"b":[{"pos":1,"start":0,"end":1,"array_positions":null},{"pos":2,"start":2,"end":3,"array_positions":null}]}},"fragments":{"description":["\u003cmark\u003eB\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e with three double rooms."]},"sort":["_score"]} +{"index":"travel-sample-index-stored_29bcd920b91f021e_aa574717","id":"hotel_26223","score":0.48263673880940405,"locations":{"description":{"b":[{"pos":83,"start":534,"end":535,"array_positions":null}],"r":[{"pos":82,"start":532,"end":533,"array_positions":null}]}},"fragments":{"description":["… fresh sushi and excellent steaks. Also located inside the hotel's lobby is the [http://www.therrazzroom.com Rrazz Room Theater], with nightly cabaret and \u003cmark\u003eR\u003c/mark\u003e\u0026\u003cmark\u003eB\u003c/mark\u003e and the Imperial Club rooms on the top fl…"]},"sort":["_score"]} +{"status":{"total":6,"failed":0,"successful":6},"request":{"query":{"regexp":"[a-z]","field":"description"},"size":10,"from":0,"highlight":{"style":"html","fields":null},"fields":null,"facets":null,"explain":false,"sort":["-_score"],"includeLocations":false},"hits":[],"total_hits":47,"max_score":2.9424485879934785,"took":980903,"facets":null} +
      • + +
    +
    diff --git a/content/sdk/c/managing-connections.dita b/content/sdk/c/managing-connections.dita index c6600dc50d..0419d7345b 100644 --- a/content/sdk/c/managing-connections.dita +++ b/content/sdk/c/managing-connections.dita @@ -6,6 +6,28 @@ This section describes how to connect the C SDK to a Couchbase cluster and bucket. It contains best practices as well as information on the connection string, SSL and other advanced connection options. + +
    + + RBAC and the SDK + + +

    Couchbase Server 5.0 Enterprise Edition introduced Role-Based Access Control (RBAC). + In order to access cluster-resources, programs, including those supported by the Couchbase SDK, must + authenticate with Couchbase Server, and so be identified as existing users, each associated with one or + more roles. Authentication requires the passing of credentials; which consist of a username + and (in most cases) a password. + Users may be defined either locally to Couchbase Server, or externally (for example, by means of an LDAP server). +

    + +

    + Once authentication has succeeded, an authorization process checks the roles associated with the + identified user. If role-associated privileges correspond to the level of resource-access requested by the user, + access is duly granted; otherwise, it is denied. +

    + +
    +
    Creating a handle

    A library handle is created using the lcb_create() function which @@ -16,6 +38,7 @@ lcb_create_st options; memset(&options, 0, sizeof options); options.version = 3; options.v.v3.connstr = "couchbase://10.3.53.2/my-bucket"; +options.v.v3.username = "jo"; options.v.v3.passwd = "s3cr3t"; lcb_error_t rc = lcb_create(&instance, &options); if (rc != LCB_SUCCESS) { @@ -27,9 +50,6 @@ if (rc != LCB_SUCCESS) { connection specification over all those other SDKs.See , which contains some additional connection string options.

    -

    If the bucket is password protected, the passwd field may be - set. Note that the bucket password (set when creating a bucket) is not the same as - the administrative password for Couchbase.

    When creating a new handle, you can also specify an alternate event loop plugin to use. This is required if you want to use your library in non-blocking mode with your existing non-blocking application.

    @@ -88,9 +108,9 @@ if (rc != LCB_SUCCESS) {

    Couchbase Sever features the ability to have clients communicate securely via SSL.

    -

    To use SSL, you need Couchbase Server Enterprise 3.0 or later

    +

    To use SSL, you need Couchbase Server Enterprise 3.0 or later (not available in the Community Edition).

      -
    1. Obtain the SSl certificate used by the Cluster
      2. +
    2. Obtain the SSL certificate used by the Cluster
    3. Make the certificate available to the file system of the client host.
    4. Employ the couchbases:// scheme for the connection string.
    5. Specify the local path to the certificate as the value for the diff --git a/content/sdk/c/n1ql-queries-with-sdk.dita b/content/sdk/c/n1ql-queries-with-sdk.dita index 42a130f40f..46e12e8789 100644 --- a/content/sdk/c/n1ql-queries-with-sdk.dita +++ b/content/sdk/c/n1ql-queries-with-sdk.dita @@ -89,6 +89,27 @@ lcb_error_t rc = lcb_n1ql_query(instance, NULL, &cmd); to application/json. Since version 2.5.3, all queries must be in JSON, and the content_type field is ignored.

      + + +
      Scan Consistency

      + Setting a staleness parameter for queries, with scan_consistency, enables a tradeoff + between latency and (eventual) consistency. A N1QL query using the default Not Bounded Scan Consistency will not wait for any + indexes to finish updating before running the query and returning results, meaning that results are returned quickly, but the + query will not return any documents yet to be indexed.

      + +

      + With Scan Consistency set to RequestPlus, all document changes and index updates are processed before + the query is run. Select this when consistency is always more important than performance. For a middle ground, + AtPlus is a "read your own write" (RYOW) option, which means it just waits for the new documents that + you specify to be indexed, rather than an entire index of multiple documents. + See the examples + for how to use AtPlus for the best performance balance for many circumstances. +

      +
      + + +
      Prepared StatementsSince version 2.5.3, applications may optimize frequently issued statements by having the client internally prepare them. To use prepared statements, simply set the LCB_CMDN1QL_F_PREPCACHE bit in the diff --git a/content/sdk/c/sample-app-backend.dita b/content/sdk/c/sample-app-backend.dita index ed8fa3ebda..a8d3586232 100644 --- a/content/sdk/c/sample-app-backend.dita +++ b/content/sdk/c/sample-app-backend.dita @@ -12,14 +12,9 @@ >json-cpp), though you are free to use your own.

      This tutorial focuses on querying through N1QL and FTS rather than - views. If you want information about using views, see the following resources:
        -
      • General Information about views: -
        • -
      • Querying views: -
        • -
      + views. If you want information about using views, see + Views. +
      Walking Through the API @@ -236,7 +231,7 @@ ORDER BY a.name ASC]]>

      Relevant Documentation Topics: , .

      In this service, hotels are searched for using more fuzzy criterias, like the content - of the address or the description of an hotel. This is done using Full Text Search + of the address or the description of a hotel. This is done using Full Text Search (FTS). When some results match the specified criteria, only the relevant data for each result to be displayed in the UI is fetched using the subdocument API.

      To find a hotel based on its location and its description, first a JSON query body is diff --git a/content/sdk/c/scan-consistency-examples.dita b/content/sdk/c/scan-consistency-examples.dita new file mode 100644 index 0000000000..6ca1167516 --- /dev/null +++ b/content/sdk/c/scan-consistency-examples.dita @@ -0,0 +1,278 @@ + + + + Scan Consistency Using the C SDK with Couchbase Server + Using Scan Consistency + You can balance performance against consistency in N1QL queries via the LCB client and the AtPlus option. + +

      + + + +

      Using CONSISTENCY_REQUEST guarantees that the query will include newly-inserted documents:#include <libcouchbase/couchbase.h> +#include <libcouchbase/api3.h> +#include <libcouchbase/n1ql.h> +#include <string> +#include <vector> + +static int RandomNumber_g; +typedef std::vector<std::string> RowList; + +static void query_callback(lcb_t, int, const lcb_RESPN1QL *resp) +{ + if (resp->rc != LCB_SUCCESS) { + fprintf(stderr, "N1QL query failed (%s)\n", lcb_strerror(NULL, resp->rc)); + } + + if (resp->rflags & LCB_RESP_F_FINAL) { + // We're simply notified here that the last row has already been returned. + // no processing needed here. + return; + } + + // Add rows to the vector, we'll process the results in the calling + // code. + RowList *rowlist = reinterpret_cast<RowList*>(resp->cookie); + rowlist->push_back(std::string(resp->row, resp->nrow)); +} + +int +main(int, char **) +{ + lcb_t instance; + lcb_create_st crst; + lcb_error_t rc; + + memset(&crst, 0, sizeof crst); + crst.version = 3; + crst.v.v3.connstr = "couchbase://10.0.0.31/default"; + + rc = lcb_create(&instance, &crst); + rc = lcb_connect(instance); + lcb_wait(instance); + rc = lcb_get_bootstrap_status(instance); + + // Initialize random seed to get a "random" value in our documents + srand(time(NULL)); + RandomNumber_g = rand() % 10000; + + char key[256]; + sprintf(key, "user:%d", RandomNumber_g); + char value[4096]; + sprintf(value, + "{" + "\"name\":[\"Brass\",\"Doorknob\"]," + "\"email\":\"brass.doorknob@juno.com\"," + "\"random\":%d" + "}", RandomNumber_g); + + printf("Will insert new document with random number %d\n", RandomNumber_g); + + lcb_CMDSTORE scmd = { 0 }; + LCB_CMD_SET_KEY(&scmd, key, strlen(key)); + LCB_CMD_SET_VALUE(&scmd, value, strlen(value)); + scmd.operation = LCB_SET; + + lcb_sched_enter(instance); + rc = lcb_store3(instance, NULL, &scmd); + lcb_sched_leave(instance); + lcb_wait(instance); + // In real code, we'd also install a store callback so we can know if the + // actual storage operation was a success. + + lcb_N1QLPARAMS *params = lcb_n1p_new(); + + lcb_CMDN1QL cmd = { 0 }; + RowList rows; + cmd.callback = query_callback; + + rc = lcb_n1p_setstmtz(params, + "SELECT name, email, random FROM default WHERE $1 IN name"); + // -1 for length indicates nul-terminated string + rc = lcb_n1p_posparam(params, "\"Brass\"", -1); + + // This guarantess that the query will include the newly-inserted document. + rc = lcb_n1p_setconsistency(params, LCB_N1P_CONSISTENCY_REQUEST); + rc = lcb_n1p_mkcmd(params, &cmd); + rc = lcb_n1ql_query(instance, &rows, &cmd); + lcb_wait(instance); + + // To demonstrate the CONSISTENCY_REQUEST feature, we check each row for the + // "random" value. Because the C standard library does not come with a JSON + // parser, we are limited in how we can inspect the row (which is JSON). + // For clarity, we print out only the row's "Random" field. When the + // CONSISTENCY_REQUEST feature is enabled, one of the results should contain + // the newest random number (the value of RandomNumber_g). When disabled, the + // row may or may not appear. + for (RowList::iterator ii = rows.begin(); ii != rows.end(); ++ii) { + std::string& row = *ii; + size_t begin_pos = row.find("\"random\""); + size_t end_pos = row.find_first_of("},", begin_pos); + std::string row_number = row.substr(begin_pos, end_pos - begin_pos); + printf("Row has random number %s\n", row_number.c_str()); + } + + lcb_n1p_free(params); + lcb_destroy(instance); +}

      +
      + +

      + Setting a Scan Consistency, lets you choose between NotBounded (the default), for sharpest performance; RequestPlus, for strongest consistency; and AtPlus, for a good balance between increased performance and completeness of results:>#include <libcouchbase/couchbase.h> +#include <libcouchbase/n1ql.h> +#include <string> +#include <vector> +#include <sstream> + +static int RandomNumber_g; +typedef std::vector<std::string> RowList; + +static void query_callback(lcb_t, int, const lcb_RESPN1QL *resp) +{ + if (resp->rc != LCB_SUCCESS) { + fprintf(stderr, "N1QL query failed (%s)\n", lcb_strerror(NULL, resp->rc)); + } + + if (resp->rflags & LCB_RESP_F_FINAL) { + // We're simply notified here that the last row has already been returned. + // no processing needed here. + return; + } + + // Add rows to the vector, we'll process the results in the calling + // code. + RowList *rowlist = reinterpret_cast<RowList*>(resp->cookie); + rowlist->push_back(std::string(resp->row, resp->nrow)); +} + +static void storage_callback(lcb_t, int cbtype, const lcb_RESPBASE *resp) +{ + if (resp->rc != LCB_SUCCESS) { + fprintf(stderr, "Failed to store document: %s\n", lcb_strerror(NULL, resp->rc)); + exit(EXIT_FAILURE); + } + + lcb_MUTATION_TOKEN *mt = reinterpret_cast<lcb_MUTATION_TOKEN*>(resp->cookie); + *mt = *lcb_resp_get_mutation_token(cbtype, resp); +} + +int +main(int, char **) +{ + lcb_t instance; + lcb_create_st crst; + lcb_error_t rc; + + memset(&crst, 0, sizeof crst); + crst.version = 3; + crst.v.v3.connstr = "couchbase://localhost/default?fetch_mutation_tokens=true"; + + rc = lcb_create(&instance, &crst); + rc = lcb_connect(instance); + lcb_wait(instance); + rc = lcb_get_bootstrap_status(instance); + + // Initialize random seed to get a "random" value in our documents + srand(time(NULL)); + RandomNumber_g = rand() % 10000; + + // Install the storage callback which will be used to retrieve the + // mutation token + lcb_install_callback3(instance, LCB_CALLBACK_STORE, storage_callback); + + lcb_MUTATION_TOKEN mt; + memset(&mt, 0, sizeof mt); + + char key[256]; + sprintf(key, "user:%d", RandomNumber_g); + char value[4096]; + sprintf(value, + "{" + "\"name\":[\"Brass\",\"Doorknob\"]," + "\"email\":\"brass.doorknob@juno.com\"," + "\"random\":%d" + "}", RandomNumber_g); + + printf("Will insert new document with random number %d\n", RandomNumber_g); + + lcb_CMDSTORE scmd = { 0 }; + LCB_CMD_SET_KEY(&scmd, key, strlen(key)); + LCB_CMD_SET_VALUE(&scmd, value, strlen(value)); + + rc = lcb_store3(instance, &mt, &scmd); + lcb_wait(instance); + + lcb_CMDN1QL cmd = { 0 }; + RowList rows; + cmd.callback = query_callback; + + // At the time of writing, the lcb_N1QLPARAMS implementation has some + // bugs in it with respect to adding mutation tokens. For this reason, we're + // encoding the query manually. This would look a bit nicer if we were + // using a real JSON library: + + const char *bucketname; + lcb_cntl(instance, LCB_CNTL_GET, LCB_CNTL_BUCKETNAME, &bucketname); + std::stringstream stmt; + + stmt << + "{" + "\"statement\":\"SELECT name, email, random FROM default WHERE $1 in name\"," + "\"args\":[\"Brass\"]," + "\"scan_consistency\":\"at_plus\","; + + stmt << "\"scan_vectors\":{" + << "\"" << bucketname << "\":{" + << "\"" < LCB_MUTATION_TOKEN_VB(&mt) < "\":[" + << LCB_MUTATION_TOKEN_SEQ(&mt) << "," + << "\"" < LCB_MUTATION_TOKEN_ID(&mt) << "\"" + << "]" + << "}" + << "}" + << "}"; + + /* + The above expands to something like this: + { + "statement": "SELECT name, email, random FROM default WHERE $1 in name", + "args": ["Brass"], + "scan_consistency": "at_plus", + "scan_vectors": { + "default": { + "29": [3, "88598346863273"] + } + } + } + */ + + std::string stmt_str = stmt.str(); + cmd.query = stmt_str.c_str(); + cmd.nquery = stmt_str.size(); + rc = lcb_n1ql_query(instance, &rows, &cmd); + assert(rc==LCB_SUCCESS); + lcb_wait(instance); + + // To demonstrate the at_plus feature, + // we check each row for the "random" value. + // Because the C standard library does not come with a JSON + // parser, we are limited in how we can inspect the row (which is JSON). + // For clarity, we print out only the row's "Random" field. When the + // at_plus feature is enabled, one of the results should contain + // the newest random number (the value of RandomNumber_g). When disabled, the + // row may or may not appear. + for (RowList::iterator ii = rows.begin(); ii != rows.end(); ++ii) { + std::string& row = *ii; + size_t begin_pos = row.find("\"random\""); + size_t end_pos = row.find_first_of("},", begin_pos); + std::string row_number = row.substr(begin_pos, end_pos - begin_pos); + printf("Row has random number %s\n", row_number.c_str()); + } + + lcb_destroy(instance); +}

      +
      + + + \ No newline at end of file diff --git a/content/sdk/c/sdk-authentication-overview.dita b/content/sdk/c/sdk-authentication-overview.dita index 2607e19685..377b0d2ac4 100644 --- a/content/sdk/c/sdk-authentication-overview.dita +++ b/content/sdk/c/sdk-authentication-overview.dita @@ -1,54 +1,54 @@ - + Authentication - + RBAC requires that users pass appropriate credentials to Couchbase Server in order to access cluster-resources. The SDK provides enhanced support for the passing of credentials. - + - +
      - -
      + +
      - +
      - + Passing Credentials - +

      - The most recent versions of the SDK are RBAC-enabled, and provide - enhanced support for authentication. Code examples for C can be found in + The most recent versions of the SDK are RBAC-enabled, and provide + enhanced support for authentication. Code examples for C can be found in Getting Started with the SDK, and also as part of the Sample Code provided to demonstrate user management.

      - +
      - +
      Upgrading to RBAC-Enabled SDK-Versions - +

      The last pre-RBAC version of the C SDK was 2.7.3. The first RBAC-enabled version was - 2.7.4. See the SDK + 2.7.4. See the SDK Release Notes for more information.

      - +

      With the most recent versions of the SDK, the legacy authentication interface and @@ -57,41 +57,41 @@ whose version is either 5.0 and beyond, or pre-5.0.

      - +
      - Further Information on RBAC + Further Information on RBAC - +

      All aspects of the Couchbase RBAC system are covered in the section Authorization. Specifically, for information on:

      - +
      • Adding Users and assigning roles, by means of - the Couchbase Web Console, see + the Couchbase Web Console, see Creating and Managing Users with the UI. - +

        - +

        • - +
      • Roles required for resource-access, and the privileges they entail, see RBAC for Applications. - +

        - +

        • - +
      • Resources controlled by Couchbase RBAC, see Resources @@ -99,7 +99,85 @@
      - + +
      + + Certificate-Based Authentication + + +

      + Couchbase Server supports the use of X.509 certificates to authenticate clients. This allows + authenticated users to access specific resources by means of the data service (no other service + is currently available to clients through this form of authentication). +

      + +

      + The process relies on a certificate + authority, for the issuing of certificates that validate identities. A certificate includes + information such as the name of the entity it identifies, an expiration date, the name of the authority + that issued the certificate, and the digital signature of the authority. + A client attempting to access Couchbase Server can present a certificate to the server, allowing + the server to check the validity of the certificate. If the certificate is valid, the user + under whose identity the client is running, + and the roles assigned that user, are verified. If the assigned roles are appropriate for + the level of access requested to the specified resource, access is granted. +

      + +

      + The C SDK has supported working with X.509 certificates since 2.8.4. Version 2.8.5 of libcouchbase added + the ability for the SSL trust store to be kept in a separate file. The trust store has to be specified + with option truststorepath=…, otherwise the library will expect it to be stored with the + certificate in certpath=, and the trust store and certificate concatenated in the certpath. +

      + +

      + This shell script will generate the client certificate - you will need to ensure that you have + OpenSSL installed on your system: + export CB_ROOT=/opt/couchbase + +export TOPDIR=SSLCA +export ROOT_DIR=rootdir +export CLIENT_DIR=clientdir +export INT_DIR=intdir +export ROOT_CA=ca +export INTERMEDIATE=int +export CLIENT=client +export CHAIN=chain + +export ADMINCRED=Administrator:password +export ip=127.0.0.1 +export USERNAME=testuser + +cd ${TOPDIR} + +mkdir -p ${CLIENT_DIR} + +cd ${CLIENT_DIR} +openssl genrsa -out ${CLIENT}.key 2048 +openssl req -new -key ${CLIENT}.key -out ${CLIENT}.csr -subj "/CN=${USERNAME}/OU=None/O=None/L=None/S=None/C=US" +openssl x509 -req -in ${CLIENT}.csr -CA ../${INT_DIR}/${INTERMEDIATE}.pem \ + -CAkey ../${INT_DIR}/${INTERMEDIATE}.key -CAcreateserial \ + -CAserial ../${INT_DIR}/intermediateCA.srl -out ${CLIENT}.pem -days 365 + +cat ./${CLIENT}.pem ../${INT_DIR}/${INTERMEDIATE}.pem ../${ROOT_DIR}/${ROOT_CA}.pem > ./${CHAIN}.pem + Which is then used by passing ${CLIENT_DIR}/${CHAIN}.pem as the certpath + option, and ${CLIENT_DIR}/${CLIENT}.key as keypath in the connection + string (and don't forget couchbases or https in the schema). For + example:couchbases://127.0.0.1?certpath=/path/to/chain.pem&keypath=/path/to/client.key

      +

      + When necessary the root certificate, along with all intermediates, might be supplied with a separate + connection string option truststorepath, and in this case concatenation (the last step + in certification generation) is not necessary, and the client certificate could be specified as is. See + full example at . +

      + The Go platform requires extra care at the certificate generation stage. If the certificate generated does not + directly match the host, then the SAN (Subject Alternative Name) Field should supply the + appropriate names: gocb may fail to authenticate if it can't verify the hostname is valid for the certificate. Other platforms + - Java and .NET, with OpenSSL based implementations - are by default less strict on verification of hostnames. +
      + - + diff --git a/content/sdk/c/sdk-user-management-overview.dita b/content/sdk/c/sdk-user-management-overview.dita index cb3929780a..411c8a30f1 100644 --- a/content/sdk/c/sdk-user-management-overview.dita +++ b/content/sdk/c/sdk-user-management-overview.dita @@ -66,8 +66,7 @@
    6. Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

      diff --git a/content/sdk/c/sdk-xattr-example.dita b/content/sdk/c/sdk-xattr-example.dita index c5597e7549..d53c7027aa 100644 --- a/content/sdk/c/sdk-xattr-example.dita +++ b/content/sdk/c/sdk-xattr-example.dita @@ -130,12 +130,12 @@ static void n1qlrow_callback(lcb_t instance, int type, const lcb_RESPN1QL *resp) char *start = "{\"docID\":\""; char *stop = "\""; - char *key = strnstr(resp->row, start, resp->nrow); + char *key = strstr(resp->row, start); if (key == NULL || key != resp->row) { return; } key += strlen(start); - char *z = strnstr(key, stop, resp->nrow - (resp->row - key)); + char *z = strstr(key, stop); if (z == NULL) { return; } diff --git a/content/sdk/c/threshold-logging.dita b/content/sdk/c/threshold-logging.dita new file mode 100644 index 0000000000..5881736754 --- /dev/null +++ b/content/sdk/c/threshold-logging.dita @@ -0,0 +1,154 @@ + + + + Threshold Logging Tracing through the SDK + Threshold Logging + + + +

      +
      + +
      +
      + +
      + + Threshold Logging in C SDK + + +

      + Response Time Observability is implemented as Threshold Logging in libcouchbase from release 2.9.0.

      + + + Threshold Logging Tracer Properties + + + + Setting name + Macro name for lcb_cntl + Description + Default Value + + + + + enable_tracing + LCB_CNTL_ENABLE_TRACING + Boolean used to determine tracing is enabled. Defaults to using the + ThesholdLoggingTracer if enabled. When false a Noop or similar tracing + implementation should be used instead. Also when enabled the SDK will try and retrieve + duration metrics from the server for KV operations. + true + + + tracing_threshold_queue_flush_interval + LCB_CNTL_TRACING_THRESHOLD_QUEUE_FLUSH_INTERVAL + The interval between executions that process the collected operation spans. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 10.0 seconds (or 10000000 microseconds) + + + tracing_threshold_queue_size + LCB_CNTL_TRACING_THRESHOLD_QUEUE_SIZE + The maximum number of items to keep in internal queue per service. + 128 + + + tracing_threshold_kv + LCB_CNTL_TRACING_THRESHOLD_KV + The KV operation operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 0.5 second (500000 microseconds) + + + tracing_threshold_view + LCB_CNTL_TRACING_THRESHOLD_VIEW + The View query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_n1ql + LCB_CNTL_TRACING_THRESHOLD_N1QL + The N1QL query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_fts + LCB_CNTL_TRACING_THRESHOLD_FTS + The FTS query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_analytics + LCB_CNTL_TRACING_THRESHOLD_ANALYTICS + The Analytics query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_orphaned_queue_flush_interval + LCB_CNTL_TRACING_ORPHANED_QUEUE_FLUSH_INTERVAL + The interval between executions that processes the collected operation spans. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 10.0 seconds (or 10000000 microseconds) + + + tracing_orphaned_queue_size + LCB_CNTL_TRACING_ORPHANED_QUEUE_SIZE + The maximum number of items to keep in internal queue per service. + 128 + + + +
      +
      + + +
      + + Customizing + +

      + You can customise Tracer settings through the connection string, or using lcb_cntl. Here + is an example on how to customize our default tracer to reduce the time interval when the information + gets logged: + +lcb_uint32_t tmoval = 10000000; /* in microseconds, 10 seconds */ +lcb_cntl(instance, LCB_CNTL_SET, LCB_CNTL_TRACING_THRESHOLD_QUEUE_FLUSH_INTERVAL, &tmoval); + + +/* the same as above, but using string-encoded float value, representing seconds with fraction part */ +lcb_cntl_string("tracing_threshold_queue_flush_interval", "10.0"); + + +/* the same, but using connection string */ +lcb_t instance; +struct lcb_create_st cropts = {0}; +lcb_error_t rc; + +cropts.version = 3; +cropts.v.v3.connstr = "couchbase://127.0.0.1/default?tracing_threshold_queue_flush_interval=10.0"; +cropts.v.v3.username = "Administrator"; +cropts.v.v3.passwd = "password"; + +rc = lcb_create(&instance, &cropts); +if (rc != LCB_SUCCESS) { + die(rc, "Creating instance"); +} + + Follow along with our example of , to discover how to use tracing in the C SDK for Response Time Observability. +

      +
      + + diff --git a/content/sdk/c/tracing-from-the-sdk.dita b/content/sdk/c/tracing-from-the-sdk.dita new file mode 100644 index 0000000000..cdc6d4c317 --- /dev/null +++ b/content/sdk/c/tracing-from-the-sdk.dita @@ -0,0 +1,136 @@ + + + + Tracing from the C SDK (libcouchbase) with Couchbase Server + Tracing from the SDK + Use tracing from the SDK to discover timeouts and bottlenecks across the network and cluster. + +

      For our example, we will customize the threshold logging tracer settings with lcb_cntl. By default it + will log every 10 seconds (if something is found) and only sample the top 10 slowest operations. The default threshold for key/value operation is 500 milliseconds. We shall + set them so low that almost everything gets logged - not something you should do in production! +lcb_U32 val = 1; +lcb_cntl(instance, LCB_CNTL_SET, LCB_CNTL_TRACING_THRESHOLD_KV, &val); /* 1 microsecond */ +lcb_cntl(instance, LCB_CNTL_SET, LCB_CNTL_TRACING_THRESHOLD_QUEUE_FLUSH_INTERVAL, &val); /* 1 second */ + +val = LCB_LOG_INFO; /* threshold logger writes at INFO level */ +lcb_cntl(instance, LCB_CNTL_SET, LCB_CNTL_CONLOGGER_LEVEL, &val); +This will set our threshold for key/value operations to one microsecond, log the found operations every second.

      + +

      With these configs in place we are ready to run some operations. Below, we read some documents from the travel-sample bucket and, if found, write them back with an upsert - giving us both read and write operations to log. +static void die(lcb_t instance, const char *msg, lcb_error_t err) +{ + fprintf(stderr, "%s. Received code 0x%X (%s)\n", msg, err, lcb_strerror(instance, err)); + exit(EXIT_FAILURE); +} + +static void op_callback(lcb_t instance, int cbtype, const lcb_RESPBASE *rb) +{ + if (rb->rc == LCB_SUCCESS && cbtype == LCB_CALLBACK_GET) { + lcb_error_t err; + lcb_CMDSTORE cmd = {0}; + const lcb_RESPGET *rg = (const lcb_RESPGET *)rb; + + LCB_CMD_SET_KEY(&cmd, rg->key, rg->nkey); + LCB_CMD_SET_VALUE(&cmd, rg->value, rg->nvalue); + cmd.operation = LCB_SET; + + err = lcb_store3(instance, NULL, &cmd); + if (err != LCB_SUCCESS) { + die(instance, "Couldn't schedule storage operation", err); + } + } +} + +int main(int argc, char *argv[]) +{ + lcb_error_t err; + lcb_t instance; + struct lcb_create_st create_options = {0}; + + create_options.version = 3; + create_options.v.v3.connstr = "couchbase://127.0.0.1/travel-sample"; + create_options.v.v3.passwd = "password"; + create_options.v.v3.username = "Administrator"; + + + err = lcb_create(&instance, &create_options); + if (err != LCB_SUCCESS) { + die(NULL, "Couldn't create couchbase handle", err); + } + + { + lcb_U32 val = 1; + lcb_cntl(instance, LCB_CNTL_SET, LCB_CNTL_TRACING_THRESHOLD_KV, &val); /* 1 microsecond */ + lcb_cntl(instance, LCB_CNTL_SET, LCB_CNTL_TRACING_THRESHOLD_QUEUE_FLUSH_INTERVAL, &val); /* 1 second */ + + val = LCB_LOG_INFO; /* threshold logger writes at INFO level */ + lcb_cntl(instance, LCB_CNTL_SET, LCB_CNTL_CONLOGGER_LEVEL, &val); + } + + err = lcb_connect(instance); + if (err != LCB_SUCCESS) { + die(instance, "Couldn't schedule connection", err); + } + + lcb_wait(instance); + + err = lcb_get_bootstrap_status(instance); + if (err != LCB_SUCCESS) { + die(instance, "Couldn't bootstrap from cluster", err); + } + + lcb_install_callback3(instance, LCB_CALLBACK_GET, op_callback); + lcb_install_callback3(instance, LCB_CALLBACK_STORE, op_callback); + + { + lcb_CMDGET cmd = {0}; + char key[] = "airline_1X"; + size_t key_len = strlen(key); + int i; + for (i = 0; i < 5; i++) { + key[key_len - 1] = i + '0'; + LCB_CMD_SET_KEY(&cmd, key, key_len); + err = lcb_get3(instance, NULL, &cmd); + if (err != LCB_SUCCESS) { + die(instance, "Couldn't schedule retrieval operation", err); + } + } + } + lcb_wait(instance); + + lcb_destroy(instance); + return 0; +} + +

      + +

      Run the code, and you will see something like the following in the logs: +22ms [I3fc6ae73] {27957} [INFO] (tracer - L:148) Operations over threshold: {"count":6,"service":"kv","top":[{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x0","last_remote_address":"192.168.1.12:11210","server_us":27,"total_us":1761},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x1","last_remote_address":"192.168.1.12:11210","server_us":8,"total_us":2189},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x2","last_remote_address":"192.168.1.12:11210","server_us":3,"total_us":2356},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x3","last_remote_address":"192.168.1.12:11210","server_us":3,"total_us":2477},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"upsert:0x5","last_remote_address":"192.168.1.12:11210","server_us":100,"total_us":2489},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x4","last_remote_address":"192.168.1.12:11210","server_us":3,"total_us":2592}]} +For each service (only kv-based on this workload), the threshold log reporter will show you the total number of recorded ops (through count), and give you the top slowest ops sorted by their latency. Since only airline_10 exists in the bucket you will see five document fetches but only one mutation.

      + + +
      + + Output fields in detail. + +

      Let's highlight a single operation, and explain each field in a little more detail: +{ + "total_us": 1761, + "server_us": 27, + "last_remote_address": "192.168.1.12:11210", + "last_operation_id": "get:0x0", + "last_local_id": "000000003fc6ae73/277953824e30913b", + "last_local_address": "192.168.1.10:56762" +} +

      +

      This tells us the following:

        +
      • total_us: The total time it took to perform the full operation: here around 1.8 milliseconds.
        • +
      • server_us: The server reported that its work performed took 27 microseconds (this does not include network time or time in the buffer before picked up at the cluster).
        • +
      • last_local_address: The local socket used for this operation.
        • +
      • last_remote_address: The remote socket on the server used for this operation. Useful to figure out which node is affected.
        • +
      • last_operation_id: A combination of type of operation and id (in this case the opaque value), useful for diagnosing and troubleshooting in combination with the last_local_id.
        • +
      • last_local_id: With Server 5.5 and later, this id is negotiated with the server and can be used to correlate logging information on both sides in a simpler fashion.

      +

      You can see that if the thresholds are set the right way based on production requirements, without much effort slow operations can be logged and pinpointed more easily than before.

      +
      + +       diff --git a/content/sdk/compression-intro.dita b/content/sdk/compression-intro.dita new file mode 100644 index 0000000000..5e17d8faba --- /dev/null +++ b/content/sdk/compression-intro.dita @@ -0,0 +1,99 @@ + + + + Compression + In response to increasing volumes of data being sent over the wire, Couchbase Data Platform now provides data compression between the SDK and Couchbase Server. + + + + + + +
      + Overview +

      Documents may already be stored compressed with Snappy. Now documents may be passed from client applications to the server in compressed form, saving around 40% in bandwidth, and also transmission time. These operations take place automatically, after HELLO negotiation with the server, and do not require any changes on the client side, beyond updating to a recent SDK version with compression support (see release notes).

      +
      + +
      + Limits +

      The document must be below 20MB in size in both compressed and uncompressed form. Compression + is only available in the Enterprise Edition of Couchbase Data Platform.

      +
      + + +
      + Operating Modes +

      The three modes in which compression can be used, “off”, “passive” and “active”, are per bucket configuration settings on the cluster. Depending on how it is set, the HELLO negotiation will fail or succeed. The HELLO flag for compression has the value 0x0a and is defined as: +PROTOCOL_BINARY_FEATURE_SNAPPY = 0x0a

      + + Compression Operating Modes + + + + + + + + + OFF + PASSIVE + ACTIVE + + + + + Sending SNAPPY feature with HELLO + ignore + acknowledge + acknowledge + + + Sending compressed data when SNAPPY feature enabled + - + accept + accept + + + Sending compressed data when SNAPPY feature disabled + reject as invalid + reject as invalid + reject as invalid + + + Receiving data which were previously compressed on the server + server inflates and sends plain data + server sends compressed data + server sends compressed data + + + Receiving data which were not previously compressed on the server + server sends plain data + server sends plain data + server might send compressed data (the compression is done in the background on the server) + + + +
      +

      + +
      + + +
      + Acceptance Guarantee +

      The server may change compression settings for the bucket at any time, but it is guaranteed that once the socket negotiates compression through HELLO, the server will never reject compressed data, even if the bucket setting has been changed.

      +
      + + +
      + Minimum Size +

      While the tiniest of documents will not be reduced in size by compressing, there is another category of slightly larger documents in whose case the time overhead of compressing and decompressing outweighs the slight advantage of marginally reduced transmission time from client to server or back.

      +

      To safeguard against the case of several thousand documents stealing CPU time to barely discernable advantage, a threshold for minimum doument size to compress is set in the SDK, with a sensible default value - that value can be seen for your chosen SDK in its API documentation (normally 32 bytes), and you can override this to disable compression:

        +
      • For Java, to disable compression from client to server, set compressionMinSize to max int (2147483647). Alternatively, to disable compression in both directions, set the Java system property com.couchbase.snappyEnabled to False.
        • +
      • For .NET, Snappy compression is not currently supported.
        • +
      • For SDKs using LCB's connstr (Node.js, PHP, Python), override the default with "compression=off" in your connection string. Additionally, for Python you can set the couchbase.bucket.Bucket.compression property to COMPRESS_NONE.
        • +
      • For Go, can also be disabled by using compression=off in the connection string.

      +
      +       +       diff --git a/content/sdk/core-operations.dita b/content/sdk/core-operations.dita index d2d32878b4..a439ad62ce 100644 --- a/content/sdk/core-operations.dita +++ b/content/sdk/core-operations.dita @@ -167,8 +167,12 @@ docid CAS=0x8234c3c0f213, Flags=0x0. Size=16 operations, by specifying one or more sections of the document to be retrievedname, email = cb.retrieve_in('user:kingarthur', 'contact.name', 'contact.email')

      7. Counters

      You can atomically - increment or decrement the numerical value of special counter document

      A - document may be used as a counter if its value is a simple ASCII number, like + increment or decrement the numerical value of special counter document

      + Do not increment or decrement counters if using XDCR. Within a single cluster the incr() is atomic, as is decr(); + across XDCR however, if two clients connecting to two different (bidirectional) + clusters issue incr concurrently, this may (and most likely will) result in the + value only getting incremented once in total. The same is the case for decr(). + A document may be used as a counter if its value is a simple ASCII number, like 42. Couchbase allows you to increment and decrement these values atomically using a special counter operation. The example below (in Python) shows how to use @@ -389,6 +393,13 @@ print repr(cb.get('binary_doc').value)Note

      When gathering resource usage statistics, note that expired-but-not-purged items (such as the expiry pager has not scanned this item yet) will still be considered with respect to the overall storage size and item count.

      + + Although the API only sets expiry values per document, it is possible that elsewhere + in the server, an expiry value is being set for every document in a bucket. Should this be the case, + the document TTL may be reduced, and the document may become unavailable to the app + sooner than expected.
      diff --git a/content/sdk/development-intro.dita b/content/sdk/development-intro.dita index d5ff07ab73..969abac3a0 100644 --- a/content/sdk/development-intro.dita +++ b/content/sdk/development-intro.dita @@ -13,7 +13,8 @@
      Using Couchbase

      Once you've installed the - server, you can start storing, retrieving, and querying documents with + server, and created + a test bucket, you can start storing, retrieving, and querying documents with Couchbase. You can start with an SDK, the command-line cbc tool, or the web browser.

      Every item in a database goes through the basic CRUD cycle, which is typical @@ -40,11 +41,11 @@ the command line client, don’t worry - we’ll walk through the steps in the next chapter.

      -
      Creating documents

      Create the document mnunberg.json. It +

      Creating documents

      Create the document josmith.json. It can be anywhere on your file system:

      { - "name": "Mark Nunberg", - "email": "mark.nunberg@couchbase.com", + "name": "Jo Smith", + "email": "jo.smith@example.com", "privs": ["admin"], "location": { "country": "United States", @@ -52,22 +53,22 @@ "city": "Reno" }, "following": [ - "u:tgreenstein", "u:ingenthr", "u:potus" + "u:asmith", "u:bsmith", "u:potus" ], - "likes": ["doge", "pastries"] + "likes": ["dogs", "pastries"] }

      Now, insert the document into Couchbase using the cbc - utility:

      $ cbc create --mode insert mnunberg < mnunberg.jsonmnunberg + utility:

      $ cbc create -u username -P password -U couchbase://hostname-or-ip/myTestBucket --mode insert josmith < josmith.jsonjosmith is the document’s ID, which is redirected (<) to the - cbc command’s standard input.
      + cbc command’s standard input.
      Reading documents by ID

      Documents can be retrieved using their IDs. Retrieving a document by ID is extremely fast. The following query takes about 1 millisecond.

      - $ cbc cat mnunberg + $ cbc cat -u username -P password -U couchbase://hostname-or-ip/josmith { - "name": "Mark Nunberg", - "email": "mark.nunberg@couchbase.com", + "name": "Jo Smith", + "email": "jo.smith@example.com", "privs": ["admin"], "location": { "country": "United States", @@ -75,28 +76,28 @@ "city": "Reno" }, "following": [ - "u:tgreenstein", "u:ingenthr", "u:potus" + "u:asmith", "u:bsmith", "u:potus" ], - "likes": ["doge", "pastries"] + "likes": ["dogs", "pastries"] }
      Reading documents by querying

      Retrieving a document may be done using a unique identifier assigned by the application at the document’s creation, or by inspecting its contents to see if it - matches a certain criteria. ID lookups are quicker, but querying documents allows + matches a certain criterion. ID lookups are quicker, but querying documents allows for richer search capabilities (for example, "Give me all likes and followed users - located in the US?" versus "Give me a user with the ID e3d882a4").

      + located in the US" versus "Give me a user with the ID e3d882a4").

      $ cbc n1ql \ 'SELECT following, likes FROM default WHERE location.country = "United States"' { "following": [ - "u:tgreenstein", - "u:ingenthr", + "u:asmith", + "u:bsmith", "u:potus" ], "likes": [ - "doge", + "dogs", "pastries" ] @@ -104,20 +105,20 @@
      Updating documents

      Updating a document means changing the existing document. For example, take the file above and edit the likes field:

      ... - "likes": ["doge", "pastries", "couchbase"] + "likes": ["dogs", "pastries", "couchbase"] ....Then use the cbc tool to update the document in - Couchbase:$ cbc create --mode replace mnunberg < mnunberg.json
      + Couchbase:$ cbc create --mode replace josmith < josmith.json
      Deleting documents

      This example shows how to delete the document with the ID - mnunberg.

      - $ cbc rm mnunberg + josmith.

      + $ cbc rm josmith
      Couchbase clients

      Clients access data by connecting to a Couchbase cluster over the network. The most - common type of client is a Couchbase SDK which is a full programmatic API that + common type of client is a Couchbase SDK, which is a full programmatic API that enables applications to take the best advantage of Couchbase. The command line client provides a quick and streamlined interface for simple access and is suitable if you just want to access an item without writing any code.

      diff --git a/content/sdk/dotnet.ditamap b/content/sdk/dotnet.ditamap index 86c09ee739..83e72e8c4e 100644 --- a/content/sdk/dotnet.ditamap +++ b/content/sdk/dotnet.ditamap @@ -19,6 +19,9 @@ + + + @@ -27,7 +30,9 @@ - + + + @@ -39,6 +44,7 @@ + @@ -50,5 +56,7 @@ + + diff --git a/content/sdk/dotnet/client-settings.dita b/content/sdk/dotnet/client-settings.dita index 77e2a9d23c..d50e30a5a4 100644 --- a/content/sdk/dotnet/client-settings.dita +++ b/content/sdk/dotnet/client-settings.dita @@ -142,6 +142,11 @@ var bucket = cluster.OpenBucket("travel-sample");

      + + AnalyticsRequestTimeout + Gets the analytics request timeout. + 75000ms + UseSsl Whether or not to use SSL encrypt data being sent to and from the @@ -155,6 +160,12 @@ var bucket = cluster.OpenBucket("travel-sample");

      enabled       11207       + + ForceSaslPlain + Forces PLAIN SASL - to work with RBAC external authentication, such as LDAP + + true + ApiPort The port to use for the Views API @@ -183,12 +194,6 @@ var bucket = cluster.OpenBucket("travel-sample");

      Views API 18092       - - OperationTimeout - The amount of time the client will wait on a pending operation before timing - out - 2500 ms - ObserveInterval The interval to wait before each Observe attempt @@ -222,15 +227,20 @@ var bucket = cluster.OpenBucket("travel-sample");

      5000 ms       - HeartbeatConfigInterval - Sets the interval for configuration "heartbeat" checks, which check for changes - in the configuration that are otherwise undetected by the client. - 10000 ms + ConfigPollEnabled* + Enables configuration heartbeat checks. + true - EnableConfigHeartBeat - Enables configuration heartbeat checks. - true + ConfigPollInterval* + Sets the interval for configuration "heartbeat" checks, + which check for changes in the configuration that are otherwise undetected by the client. + 2500ms + + + ConfigPollCheckFloor* + Gets or sets the heartbeat configuration check floor - the minimum time between config checks. + 50ms DefaultConnectionLimit @@ -346,9 +356,50 @@ var bucket = cluster.OpenBucket("travel-sample");

      Gets or sets the search request timeout. 75000ms       + + CertificateFactory + Factory for retrieving X509 certificates from a store, or from the file + system. + null + + + ConfigurationProviders + Control which server configuration providers are used to bootstrap the cluster + and monitor for cluster changes. + CarrierPublication and HttpStreaming + + + IgnoreRemoteCertificateNameMismatch + + Static. If TLS/SSL is enabled via UseSsl, setting this to true will disable + hostname validation when authenticating connections to Couchbase Server. This is typically + done in test or development enviroments where a domain name (FQDN) has not been specified + for the bootstrap server's URI, and the IP address is used to validate the certificate, + which will fail with a RemoteCertificateNameMismatch error. + False + + + KvServerCertificateValidationCallback + Gets or sets the SSL validation callback for K/V to override the default callback. + Null + + + HttpServerCertificateValidationCallback + Gets or sets the SSL validation callback for HTTP Services (N1QL, Analytics, Views, etc.) to + override the default callback. + Null + + + UseInterNetworkV6Addresses + Gets or sets a value indicating whether to use IP version 6 addresses. + falseSearchRequestTimeout + +

      * ConfigPollEnabled, ConfigPollInterval, and ConfigPollCheckFloor + replace the previously used, deprecated settings + of, respectively, EnableConfigHeartBeat, HeartbeatConfigInterval, and HeartbeatConfigCheckFloor.

      BucketConfiguration Settings @@ -456,7 +507,7 @@ var bucket = cluster.OpenBucket("travel-sample");

      WaitTimeout The amount of time to wait for a TCP connection before timing out after the MaxSize has been reached and all connections are being - used

      Note: not used with Multiplexing IO Service

       + used

      Note: not used with Multiplexing IO Service - SSL only

      2500 ms @@ -543,19 +594,29 @@ var bucket = cluster.OpenBucket("travel-sample");

      configure the IOService either through the configuration file (app.config, web.config or json settings file) or programmatically using the SDK's ClientConfiguration when initializing the Cluster.

      -

      Multiplexing IO Service

      +

      Shared-Pooled IO Service w/MUX

      +

      As of v2.5.0, the client uses a Shared-Pooled IO Service for K/V along with Multiplexing + (MUX) by default. A shared pool allows multiple threads to use a connection at the same time + and each thread is assigned a connection from the pool in a round-robin fashion. This gives + the best performance and the benefits of both Multiplexing IO Service and Pooled IO Service. + This is the reccomended configuration and is the default from v2.5.0 on - there is no need + for additional configuration to use it.

      +

      Multiplexing IO Service (Obsolete)

      Multiplexing is when a single TCP connection is used by multiple threads to send and receive requests simultaneously. The benefit's are improved throughput with less strain on the OS resources and importantly, a reduction on the number of TCP connections between the client and the server. The Multiplexing IO Service is the default IO Service if no custom configuration is provided.

      -

      Pooled IO Service

      +

      Pooled IO Service (SSL only)

      Pooled is when the SDK maintains a pool of TCP sockets and each thread uses a single socket at one time for Memcached K/V operations (Get, Insert, Remove, etc.). The connection pool limit is capped by the MinSize and MaxSize configuration settings; if MaxSize has been reached, the SDK would wait for a connection to become available or it would return an OperationTimeout response.

      -

      Configuring the IO Service

      +

      Configuring the IO ServiceThe default IO Service as of v2.6.0 is Shared-Pooled + IO Service w/MUX and is the reccomended IO Service to use for non-SSL/TLS. We do not + recommend using any of the other IO services or changing the default settings for IO + Services as described below.

      To configure the IO Service in your configuration file you would set the connectionPool element and ioService elements. An example of setting the configuration file to use PooledIOService would look like @@ -579,8 +640,8 @@ var bucket = cluster.OpenBucket("travel-sample");

      To configure the IO Service programmatically you would use the ClientConfiguration and - setting the ConnectionPoolCreator and IOServiceCreator properties. An example that - configures the Multiplexing IO Service would look like + setting the IOServiceCreator properties. An example that configures the Multiplexing IO + Service would look like this:var config = new ClientConfiguration { Servers = newList<Uri> @@ -588,7 +649,7 @@ var bucket = cluster.OpenBucket("travel-sample");

      new Uri(ConfigurationManager.AppSettings["bootstrapUrl"]) } }; -config.ConnectionPoolCreator = ConnectionPoolFactory.GetFactory<ConnectionPool<MultiplexingConnection>>(); +config = ConnectionPoolFactory.GetFactory<ConnectionPool<MultiplexingConnection>>(); config.IOServiceCreator = IOServiceFactory.GetFactory<MultiplexingIOService>(); ClusterHelper.Initialize(config);

      Caveats

      @@ -604,6 +665,10 @@ ClusterHelper.Initialize(config);

    7. WaitTimeout
    8. MUX has no effect on Views, N1QL or Full Text Search queries
      9. +
    9. Shared-Pooled IO Service w/MUX currently ignores the MinSize and creates the MaxSize of + connections configured. This is a known issue/limitation and can be tracked by the + following Jira ticket:
    diff --git a/content/sdk/dotnet/collecting-information-and-logging.dita b/content/sdk/dotnet/collecting-information-and-logging.dita index 286c7e8150..d983d91292 100644 --- a/content/sdk/dotnet/collecting-information-and-logging.dita +++ b/content/sdk/dotnet/collecting-information-and-logging.dita @@ -4,24 +4,32 @@ Collecting Information and Logging in the .NET SDK with Couchbase Server Collecting Information The logging infrastructure in the .NET SDK depends on which .NET runtime you're - targerting. For applications that are targeting .NET Framework 4.5+ the SDK uses the Apache + targerting. For applications that are targeting .NET Framework 4.5.2+ the SDK uses the Apache Common Infrastructure Libraries and applications that are targeting .NET Core the SDK uses Microsoft.Extensions.Logging. -
    Logging in applications targeting .NET Framework 4.5+ +
    Logging in applications targeting .NET Framework 4.5.2+

    The Apache Common Infrstucture libraries provides a common interface for logging adapters. - The SDK currently uses version 3.3.1 of Common.Logging.

    + The SDK currently uses version 3.4.1 of Common.Logging for versions Couchbase .NET SDK + v2.6.0 and greater.

    In order to use logging within the SDK, you need to reference the packages for the adapter you want to use. There are adapters available for all the major logging implementations. More information and a list of available adapters is available on the project website here.

    +

    The recommended dependencies for Couchbase .NET SDK v2.6.0 greater are:

      +
    • Log4Net v2.07
      • +
    • Common.Logging 3.4.1
      • +
    • Common.Logging.Core 3.4.1
      • +
    • Common.Logging.Log4Net207 v3.4.1
      • +

    The following steps describe the process for using the Log4Net Adapter with the SDK:

    1. Using the NuGet Package Manager in Visual Studio, include the following package in - your project: Common.Logging.Log4Net1213 - + your project: Common.Logging.Log4Net207

    2. In your App.Config or Web.Config, add the following elements between @@ -38,7 +46,7 @@ <common> <logging> - <factoryAdapter type="Common.Logging.Log4Net.Log4NetLoggerFactoryAdapter, Common.Logging.Log4Net1213"> + <factoryAdapter type="Common.Logging.Log4Net.Log4NetLoggerFactoryAdapter, Common.Logging.Log4Net207"> <arg key="configType" value="INLINE" /> </factoryAdapter> </logging> @@ -65,111 +73,200 @@

    -
    Logging in applications targeting .NET Core -

    As part of the new .NET Core runtime, Microsoft have added a new abstraction interface. This - is conceptually very similar to the Common.Logging, however it also has additional support built in - for ASP.NET 5 and other Microsoft libraries.

    -

    To get logging to work within the .NET SDK, you need will need to add a package for the - adapter to connect with. There are a number of logging implementations based on - Microsoft.Extensions.Logging that can be viewed here. There - are also some basic Microsoft provided adapters such as Debug, Console and EventLog.

    -

    The following instructions show how to get up and running with the NLog adapter with the - SDK in an ASP.NET 5 web application using the newer Visual Studio 2017 format:

      -
    1. Add the following two nuget dependencies: CouchbaseNetClient and - NLog.Web.AspNetCore.
      2. -
    2. Create a nlog.config in the root of your project and enable copy to output - director. An example config could look like - this:<?xml version="1.0" encoding="utf-8" ?> -<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - autoReload="true" - internalLogLevel="Warn" - internalLogFile="c:\temp\internal-nlog.txt"> - - <!-- Load the ASP.NET Core plugin --> - <extensions> - <add assembly="NLog.Web.AspNetCore"/> - </extensions> - - <!-- the targets to write to --> - <targets> - <!-- write logs to file --> - <target xsi:type="File" name="allfile" fileName="c:\temp\nlog-all-${shortdate}.log" - layout="${longdate}|${event-properties:item=EventId.Id}|${logger}|${uppercase:${level}}|${message} ${exception}" /> - - <!-- another file log, only own logs. Uses some ASP.NET core renderers --> - <target xsi:type="File" name="ownFile-web" fileName="c:\temp\nlog-own-${shortdate}.log" - layout="${longdate}|${event-properties:item=EventId.Id}|${logger}|${uppercase:${level}}| ${message} ${exception}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" /> - - <!-- couchbase log file --> - <target xsi:type="File" name="couchbase" fileName="c:\temp\couchbase-${shortdate}.log" - layout="${longdate}|${event-properties:item=EventId.Id}|${logger}|${uppercase:${level}}| ${message} ${exception}" /> - - <!-- write to the void aka just remove --> - <target xsi:type="Null" name="blackhole" /> - </targets> - - <!-- rules to map from logger name to target --> - <rules> - <!--All logs, including from Microsoft--> - <logger name="*" minlevel="Trace" writeTo="allfile" /> - - <!--Skip Microsoft logs and so log only own logs--> - <logger name="Microsoft.*" minlevel="Trace" writeTo="blackhole" final="true" /> - <logger name="Couchbase.*" minlevel="Trace" writeTo="couchbase" final="true" /> - <logger name="*" minlevel="Trace" writeTo="ownFile-web" /> - </rules> -</nlog> -
      3. -
    3. in startup.cs update both Startup and Configure methods to look like the - following:public Startup(IHostingEnvironment env) -{ - var builder = new ConfigurationBuilder() - .SetBasePath(env.ContentRootPath) - .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true) - .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true) - .AddEnvironmentVariables(); - Configuration = builder.Build(); - - // register and load NLog config - env.ConfigureNLog("nlog.config"); -} - +
      Logging in applications targeting .NET Core +

      As part of the new .NET Core runtime, Microsoft have added a new abstraction interface. This + is conceptually very similar to the Common.Logging, however it also has additional support built in + for ASP.NET 5 and other Microsoft libraries.

      +

      To get logging to work within the .NET SDK, you need will need to add a package for the + adapter to connect with. There are a number of logging implementations based on + Microsoft.Extensions.Logging that can be viewed here. There + are also some basic Microsoft provided adapters such as Debug, Console and EventLog.

      +

      Using NLog in a Console Application

      +

      The following instructions show you how to get up and running with NLog in a Console + Application with the Couchbase .NET SDK.

      +

      Dependencies:

        +
      • CouchbaseNetClient v2.6.0 or greater
        • +
      • NLog.Extensions.Logging 1.2.0
        • +

      +

      1. Add the dependencies above using the NuGet Package Manager:

      +

      2. Copy the following nlong.config file into your project and set "Copy to output + directory" to "Copy always" in Visual Studio:

      +

      + + + + + + + + + + + + + + + + + ]]>

      +

      Add the following code to the class that contains your entrypoint into the + application:

      +

      + + static void Main(string[] args) + { + ILoggerFactory loggerFactory = new LoggerFactory(); + loggerFactory.AddNLog(new NLogProviderOptions + { + CaptureMessageTemplates = true, + CaptureMessageProperties = true + }); + NLog.LogManager.LoadConfiguration("nlog.config"); + + var config = new ClientConfiguration + { + LoggerFactory = loggerFactory, + Servers = new List<Uri> + { + new Uri("http://http://localhost:8091") + } + }; + + var cluster = new Cluster(config); + cluster.Authenticate("Administrator", "password"); + + var bucket = cluster.OpenBucket("memcached"); + + var insert = bucket.Insert("mykey", "mydoc"); + Console.WriteLine(insert.Status); + + Console.Read(); + } +

      Using NLog in a Web Application

      +

      The following instructions show how to get up and running with the NLog adapter with the + SDK in an ASP.NET 5 web application using the newer Visual Studio 2017 format:

        +
      1. Add the following two nuget dependencies: CouchbaseNetClient 2.6.X and + NLog.Web.AspNetCore 4.5.4.

        2. +
      2. Create a nlog.config in the root of your project and set "Copy to output + directory" to "Copy always" in Visual Studio. An example config could look like + this: + + + + + + + + + + + + + + + + + + + + + + + + + +]]>
        3. +
      3. In Program.cs add the following code to the Main() + method: + using Microsoft.Extensions.Logging; + using NLog.Web; + + public class Program + { + public static void Main(string[] args) + { + + // NLog: setup the logger first to catch all errors + var logger = NLog.Web.NLogBuilder.ConfigureNLog("nlog.config").GetCurrentClassLogger(); + try + { + logger.Debug("init main"); + BuildWebHost(args).Run(); + } + catch (Exception ex) + { + //NLog: catch setup errors + logger.Error(ex, "Stopped program because of exception"); + throw; + } + finally + { + // Ensure to flush and stop internal timers/threads before application-exit (Avoid segmentation fault on Linux) + NLog.LogManager.Shutdown(); + } + } + + public static IWebHost BuildWebHost(string[] args) => + WebHost.CreateDefaultBuilder(args) + .UseStartup<Startup>() + .ConfigureLogging(logging => + { + logging.ClearProviders(); + logging.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Trace); + }) + .UseNLog() // NLog: setup NLog for Dependency injection + .Build(); + }
        4. +
      4. In startup.cs update the Startup method to look like the following:
        5. -
      5. Inject a ILogger and write a log using the Microsoft Logger interface, for - example:public class HomeController : Controller -{ - private readonly ILogger _logger; + app.UseStaticFiles(); - public HomeController(ILogger logger) + app.UseMvc(routes => { - _logger = logger; - } + routes.MapRoute( + name: "default", + template: "{controller=Home}/{action=Index}/{id?}"); + }); - public IActionResult Index() + var config = new ClientConfiguration { - _logger.LogInformation("Index page says hello"); - return View(); - } -} -
        6. -

      -

      More details on configuring NLog with Microsoft.Extensions.Logging can be found here.

      -
      - + LoggerFactory = loggerFactory, + Servers = new List + { + new Uri("http://localhost:8091") + } + }; + + ClusterHelper.Initialize(config, new PasswordAuthenticator("Administrator", "password")); +}]]> +
      4. + +

    +

    More details on configuring NLog with Microsoft.Extensions.Logging can be found here.

    +
    + diff --git a/content/sdk/dotnet/compatibility-versions-features.dita b/content/sdk/dotnet/compatibility-versions-features.dita index 84c65b37b5..3ebcf99af2 100644 --- a/content/sdk/dotnet/compatibility-versions-features.dita +++ b/content/sdk/dotnet/compatibility-versions-features.dita @@ -14,7 +14,7 @@
  • ✖ Unsupported: This combination is not tested, and is not within the scope of technical support if you have purchased a support agreement.
  • ◎ Compatible: This combination has been tested previously, and should be - compatible. This combination is not supported or recommended by our technical support + compatible. This combination is not recommended by our technical support organization. It is best to upgrade either the SDK or the Couchbase version you are using.
  • ✔ Supported:This combination is subject to ongoing quality assurance, and is @@ -28,99 +28,30 @@ - - - CB/SDK - SDK 1.3 SDK 2.0 - SDK 2.1 - SDK 2.2 - SDK 2.3 - SDK 2.4 + SDK 2.1, SDK 2.2, SDK 2.3, SDK 2.4 SDK 2.5 - - CB 2.5 - ◎ - ✖ - ✖ - ✖ - ✖ - ✖ - ✖ - - - CB 3.0 - ◎ - ◎ - ◎ - ◎ - ◎ - ◎ - ◎ - CB 3.1 ◎ ✔ ✔ - ✔ - ✔ - ✔ - ✔ - - - CB 4.0 - ✖ - ✔ - ✔ - ✔ - ✔ - ✔ - ✔ - CB 4.1 - ✖ - ✔ - ✔ - ✔ - ✔ - ✔ - ✔ - - - CB 4.5 - ✖ - ✔ - ✔ - ✔ - ✔ - ✔ - ✔ - - - CB 4.6 - ✖ - ✔ - ✔ - ✔ - ✔ + CB 4.0 - 4.6 + ◎ ✔ ✔ - CB 5.0 - ✖ - ✔ - ✔ - ✔ - ✔ + CB 5.0 - 5.1 + ◎ ✔ ✔ @@ -218,7 +149,7 @@ - Full text Search + Full Text Search Not Supported Since .NET SDK 2.3 diff --git a/content/sdk/dotnet/encrypting-using-sdk.dita b/content/sdk/dotnet/encrypting-using-sdk.dita new file mode 100644 index 0000000000..6172cc9210 --- /dev/null +++ b/content/sdk/dotnet/encrypting-using-sdk.dita @@ -0,0 +1,116 @@ + + + + Field Level Encryption from the .NET SDK + Field Level Encryption is available in Couchbase Data Platform 5.5, from .NET SDK version 2.6.0. + +
    + Packaging +

    The Couchbase .NET SDK uses .NET Attributes to declaratively define which fields will be encrypted on a POCO (Plain Old CSharp Object) that matches the structure of your JSON document.

    +

    Field Level Encryption (FLE) for .NET is available as a NuGet package for download. The name of the package is Couchbase.Extensions.Encryption. Using the NuGet Package Manager, it can be installed using the following command:Install-Package Couchbase.Extensions.Encryption -Version 1.0.0-dp1

    + +

    If you are using the .NET CLI, then it can be installed using this command:dotnet add package Couchbase.Extensions.Encryption --version 1.0.0-dp1

    + +

    If you are using Visual Studio, just type Couchbase.Extensions.Encryption into the NuGet Package Manager search box; locate the package, and install it. + + Installing Couchbase.Extensions.Encryption via NuGET + + + This will also include the dependency on the Couchbase .NET SDK, which is required by the Field Level Encryption extension.

    +
    + + +
    + Configuring Field Level Encryption +

    Once you have included the FLE package in your Visual Studio or CodeVS project, you’ll need to configure the SDK to use it. FLE + currently doesn’t allow for config file configuration; this has to be done programmatically using the ClientConfiguration + found in the Couchbase.Configuration.Client namespace. Note that all of code snippets below can be found in full in + our devguide-examples repo + in GitHub.

    + +

    First, add the following namespaces to your code file:using System.Collections.Generic; +using Couchbase.Configuration.Client; +using Couchbase.Extensions.Encryption; +using Couchbase.Extensions.Encryption.Providers; +using Couchbase.Extensions.Encryption.Stores;

    + +

    Then, add the configuration required to enable FLE to use the SDK:const string publicKey = "!mysecretkey#9^5usdk39d&dlf)03sL"; +const string publicKeyName = "publickey"; + +const string signingKey = "myauthpassword"; +const string signingKeyName = "mysecret"; + +var keystore = new InsecureKeyStore( + new KeyValuePair<string, string>(publicKeyName, publicKey), + new KeyValuePair<string, string>(signingKeyName, signingKey)); + +var cryptoProvider = new AesCryptoProvider(keystore) +{ + PublicKeyName = publicKeyName, + SigningKeyName = signingKeyName +}; + +var config = new ClientConfiguration(); +config.EnableFieldEncryption(cryptoProvider); +var cluster = new Cluster(config);

    + +

    First the keystore is created with a public key and a signing key; the signing key is required for ensuring that the encrypted data or any part + of the message is not tampered with. Then an AesCryptoProvider instance is created which takes the keystore as a constructor parameter. Finally + the configuration is created and the extension method EnableFieldEncryption is called passing in the ICryptoProvider instance. Once this is done, + configuration is complete! +

    + +

    Note, that this example uses the InsecureKeyStore class which is an in-memory keystore that stores the keys unprotected; this is fine for testing and examples, but for real world applications the FileSystemKeyStore, which uses the DAPI, is a better choice.

    + +

    Once we have setup our configuration, we’ll open a cluster and bucket object. Note that for a real-world application, you’ll want to tie the scope of both the cluster and the bucket to the lifespan of the application - see the Managing Connections documentation for more information.

    +
    + + +
    + Specifying what to Encrypt +

    In order to specify which field to encrypt, a special .NET attribute is used to annotate a property on a POCO (Plain Old Csharp Object) For cross-sdk support, although any property on a POCO can be encrypted, this must be a root-level property.public class Person +{ + [EncryptedField(Provider = "AES-256-HMAC-SHA256")] + public string Password { get; set; } + public string FirstName { get; set; } + public string LastName { get; set; } + public string UserName { get; set; } + public int Age { get; set; } +}

    + +

    In the POCO above, we have a field which we wish to encrypt called Password which represents a person’s password. This + property is annotated with EncryptedFieldAttribute and the ICryptoProvider.Provider name we are using. + Note, this must match the type of ICryptoProvider we supplied in our configuration (“AES-256-HMAC-SHA256"). +

    +
    + + +
    + Storing and Retrieving Documents with Encrypted Fields +

    Once this is done, it's simply a matter of using the basic CRUD operations that Couchbase supports to insert or fetch the document from the database. + For example, assuming a Person class annotated with the EncryptedField attribute on a Password property:var teddy = new Person +{ + Age = 33, + FirstName = "Ted", + LastName = "DeBloss", + Password = "ssloBeD12345" +}; + +var bucket = cluster.OpenBucket(); +var insert = bucket.Upsert("person::1", teddy);

    + +

    Retrieving a document with an encrypted field is no different than any other get assuming you use an annotated POCO:var get = bucket.Get<Person>("person::2"); +if (get.Success) +{ + Console.WriteLine("Fetched person..."); +}

    + +

    It's important to note that the data is encrypted for transport and storage only. When you retrieve the document the value for Password + will be in its unencrypted format.

    +
    + + + +     +     diff --git a/content/sdk/dotnet/full-text-searching-with-sdk.dita b/content/sdk/dotnet/full-text-searching-with-sdk.dita index d3e78ece3c..273b09a49f 100644 --- a/content/sdk/dotnet/full-text-searching-with-sdk.dita +++ b/content/sdk/dotnet/full-text-searching-with-sdk.dita @@ -86,5 +86,267 @@ foreach (var result in results)

    We then call the CouchbaseBucket.Query(SearchQuery query) method and iterate over the ISearchQueryResult object which will gives the results of the query.

    + +
    + + Detailed Example + + +

    + The following example demonstrates Full Text Search queries that + can be made with the .NET SDK. +

    + + +static void Main(string[] args) +{ + // Access the cluster that is running on the local host, authenticating with + // the username and password of any user who has the "FTS Searcher" role + // for the "travel-sample" bucket... + // + + var config = new ClientConfiguration + { + Servers = new List<Uri> {new Uri("couchbase://localhost")} + }; + + using (var cluster = new Cluster(config)) + { + cluster.Authenticate("Administrator", "password"); + + var travelSample = cluster.OpenBucket("travel-sample"); + SimpleTextQuery(travelSample); + SimpleTextQueryOnStoredField(travelSample); + SimpleTextQueryOnNonDefaultIndex(travelSample); + TextQueryOnStoredFieldWithFacet(travelSample); + DocIdQueryMethod(travelSample); + UnAnalyzedTermQuery(travelSample, 0); + UnAnalyzedTermQuery(travelSample, 2); + MatchPhraseQueryOnStoredField(travelSample); + UnAnalyzedPhraseQuery(travelSample); + ConjunctionQueryMethod(travelSample); + QueryStringMethod(travelSample); + WildCardQueryMethod(travelSample); + NumericRangeQueryMethod(travelSample); + RegexpQueryMethod(travelSample); + } +} + +public static void SimpleTextQuery(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-unstored", + Query = new MatchQuery("swanky") + } + .Limit(10); + var result = bucket.Query(query); + + PrintResult("Simple Text Query", result); +} + +public static void SimpleTextQueryOnStoredField(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-stored", + Query = new MatchQuery("MDG").Field("destinationairport") + } + .Limit(10); + + var result = bucket.Query(query); + + PrintResult("Simple Text Query on Stored Field", result); +} + +public static void SimpleTextQueryOnNonDefaultIndex(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-hotel-description", + Query = new MatchQuery("swanky") + } + .Limit(10); + + var result = bucket.Query(query); + + PrintResult("Simple Text Query on Non-Default Index", result); +} + +public static void TextQueryOnStoredFieldWithFacet(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-stored", + Query = new MatchQuery("La Rue Saint Denis!!").Field("reviews.content") + } + .Limit(10) + .Highlighting(HighLightStyle.Ansi) + .Facets(new TermFacet("Countries Referenced", "country", 5)); + + var result = bucket.Query(query); + + PrintResult("Match Query with Facet, Result by Row", result); + + Console.WriteLine(); + Console.WriteLine("Match Query with Facet, Result by hits:"); + Console.WriteLine(result.Hits); + + Console.WriteLine(); + Console.WriteLine("Match Query with Facet, Result by facet: "); + Console.WriteLine(result.Facets); +} + +public static void DocIdQueryMethod(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-unstored", + Query = new DocIdQuery("hotel_26223", "hotel_28960") + }; + + var result = bucket.Query(query); + + PrintResult("DocId Query", result); +} + +public static void UnAnalyzedTermQuery(IBucket bucket, int fuzzinessLevel) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-stored", + Query = new TermQuery("sushi").Field("reviews.content").Fuzziness(fuzzinessLevel) + } + .Limit(50) + .Highlighting(HighLightStyle.Ansi); + + var result = bucket.Query(query); + + PrintResult("Unanalyzed Term Query with Fuzziness Level of " + + fuzzinessLevel + ":", result); +} + +public static void MatchPhraseQueryOnStoredField(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-stored", + Query = new MatchQuery("Eiffel Tower").Field("description") + } + .Limit(10) + .Highlighting(HighLightStyle.Ansi); + + var result = bucket.Query(query); + + PrintResult("Match Phrase Query, using Analysis", result); +} + +public static void UnAnalyzedPhraseQuery(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-stored", + Query = new PhraseQuery("dorm", "rooms").Field("description") + } + .Limit(10) + .Highlighting(HighLightStyle.Ansi); + + var result = bucket.Query(query); + + PrintResult("Phrase Query, without Analysis", result); +} + +public static void ConjunctionQueryMethod(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-stored", + Query = new ConjunctionQuery( + new MatchQuery("La Rue Saint Denis!!").Field("reviews.content"), + new MatchQuery("boutique").Field("description") + ) + } + .Limit(10) + .Highlighting(HighLightStyle.Ansi); + + var result = bucket.Query(query); + + PrintResult("Conjunction Query", result); +} + +public static void QueryStringMethod(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-unstored", + Query = new QueryStringQuery("description: Imperial") + } + .Limit(10); + + var result = bucket.Query(query); + + PrintResult("Query String Query", result); +} + +public static void WildCardQueryMethod(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-stored", + Query = new WildcardQuery("bouti*ue").Field("description") + } + .Limit(10) + .Highlighting(HighLightStyle.Ansi); + + var result = bucket.Query(query); + + PrintResult("Wild Card Query", result); +} + +public static void NumericRangeQueryMethod(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-unstored", + Query = new NumericRangeQuery().Min(10100).Max(10200).Field("id") + } + .Limit(10); + + var result = bucket.Query(query); + + PrintResult("Numeric Range Query", result); +} + +public static void RegexpQueryMethod(IBucket bucket) +{ + var query = new SearchQuery + { + Index = "travel-sample-index-stored", + Query = new RegexpQuery("[a-z]").Field("description") + } + .Limit(10) + .Highlighting(HighLightStyle.Ansi); + + var result = bucket.Query(query); + + PrintResult("Regexp Query", result); +} + +private static void PrintResult(string label, ISearchQueryResult resultObject) +{ + Console.WriteLine(); + Console.WriteLine("= = = = = = = = = = = = = = = = = = = = = = ="); + Console.WriteLine("= = = = = = = = = = = = = = = = = = = = = = ="); + Console.WriteLine(); + Console.WriteLine(label); + Console.WriteLine(); + + foreach (var row in resultObject) + { + Console.WriteLine(row); + } +} + +
    diff --git a/content/sdk/dotnet/images/common-logging-log4net207.jpg b/content/sdk/dotnet/images/common-logging-log4net207.jpg new file mode 100644 index 0000000000..7cae6c54a0 Binary files /dev/null and b/content/sdk/dotnet/images/common-logging-log4net207.jpg differ diff --git a/content/sdk/dotnet/images/logging-nuget-nlog-web-aspnet.PNG b/content/sdk/dotnet/images/logging-nuget-nlog-web-aspnet.PNG new file mode 100644 index 0000000000..53c31d3135 Binary files /dev/null and b/content/sdk/dotnet/images/logging-nuget-nlog-web-aspnet.PNG differ diff --git a/content/sdk/dotnet/images/logging-nuget-nlog-web-aspnet.png b/content/sdk/dotnet/images/logging-nuget-nlog-web-aspnet.png new file mode 100644 index 0000000000..53c31d3135 Binary files /dev/null and b/content/sdk/dotnet/images/logging-nuget-nlog-web-aspnet.png differ diff --git a/content/sdk/dotnet/images/logging-nuget-nlog.PNG b/content/sdk/dotnet/images/logging-nuget-nlog.PNG new file mode 100644 index 0000000000..5f0adce739 Binary files /dev/null and b/content/sdk/dotnet/images/logging-nuget-nlog.PNG differ diff --git a/content/sdk/dotnet/images/logging-nuget-nlog.png b/content/sdk/dotnet/images/logging-nuget-nlog.png new file mode 100644 index 0000000000..5f0adce739 Binary files /dev/null and b/content/sdk/dotnet/images/logging-nuget-nlog.png differ diff --git a/content/sdk/dotnet/managing-clusters.dita b/content/sdk/dotnet/managing-clusters.dita index ba3642151a..b9f1e88e3f 100644 --- a/content/sdk/dotnet/managing-clusters.dita +++ b/content/sdk/dotnet/managing-clusters.dita @@ -156,9 +156,12 @@ bucketManager.InsertDesignDocument("landmarks", designDoc); installing Puppet, Vagrant and VirtualBox can be found here: http://nitschinger.at/A-Couchbase-Cluster-in-Minutes-with-Vagrant-and-Puppet

    -

    Once the nodes are provisioned, we can continue with the rest of the example.

    +

    Should you need a .Net Core compatible DNS SRV mechanism for discovering a Couchbase cluster dynamically, + + follow the instructions at Couchbase Labs.

    +

    Once the nodes are provisioned, we can continue with the rest of the example.

    -

    Using the IP's of your nodes that have been provisioned, the client can bootstrap to the +

    Using the IP addresses of your nodes that have been provisioned, the client can bootstrap to the entry-point or "EP" node. This done either through a configuration object or through the App.config or Web.config (for ASP.NET projects). In this example we will use a ClientConfiguration object to programmatically create a configuration and bootstrap to the EP diff --git a/content/sdk/dotnet/managing-connections.dita b/content/sdk/dotnet/managing-connections.dita index c7355cf672..7a38ebd011 100644 --- a/content/sdk/dotnet/managing-connections.dita +++ b/content/sdk/dotnet/managing-connections.dita @@ -14,16 +14,25 @@ initialized, followed by one or more calls to OpenBucket() or GetBucket(..):

    -

    The ClusterHelper is a singleton/multiton that makes it easier to manage resources - and instances in server runtime environments such as ASP.NET and Owin/Katana. A - ClusterHelper will make a singleton instance of a Cluster object and configuration and - store references to opened IBucket objects that can be reused throughout the lifespan of - an application.If no further arguments are applied, this code connects to +

    The ClusterHelper is a singleton/multiton that makes it easier to + manage resources and instances in server runtime environments such as ASP.NET and + Owin/Katana. A ClusterHelper will make a singleton instance of a + Cluster object and configuration and store references to opened + IBucket objects that can be reused throughout the lifespan of an + application.The PasswordAuthenticator is used for + connecting to Couchbase Server 5.0 and greater using RBAC (Role Based Access Control). In + this case we are using the administrator account, however, a more secure way to connect + is to create a user with minimal privileges and a unique password. If you are + connecting to an older server version that does not use RBAC then you would omit the + PasswordAuthenticator and instead pass the bucket password into the OpenBucket + method.If no further arguments are applied, this code connects to localhost and opens the default bucket. While this is suitable for development, you most probably want to connect to a remote cluster and also a different bucket (with a password), to do this you create a ClientConfiguration @@ -36,13 +45,17 @@ var config = new ClientConfiguration { } }; -//create the cluster and open the new bucket +//create the cluster and pass in the RBAC user var cluster = new Cluster(config); -var bucket = cluster.OpenBucket("mybucket", "mybucketpassword"); +var credentials = new PasswordAuthenticator("Administrator", "password"); +cluster.Authenticate(credentials); + +//open the new bucket +var bucket = cluster.OpenBucket("mybucket"); //or alternately -ClusterHelper.Initialize(config); -var bucket = ClusterHelper.GetBucket("mybucket", "mybucketpassword");]]> +ClusterHelper.Initialize(config, credentials); +var bucket = ClusterHelper.GetBucket("mybucket");]]>

    The ClientConfiguration object also exposes per bucket configuration settings, for example:

    {{ "mybucket", new BucketConfiguration{ - BucketName = "mybucket", - Password = "mybucketpassword" + BucketName = "mybucket" } } } }; +//create the authenticator for passing in the user and password for RBAC +var credentials = new PasswordAuthenticator("Administrator", "password"); //create the cluster and open the new bucket -var cluster = new Cluster(config); +var cluster = new Cluster(config, credentials); var bucket = cluster.OpenBucket("mybucket"); //or alternately -ClusterHelper.Initialize(config); +ClusterHelper.Initialize(config, credentials); var bucket = ClusterHelper.GetBucket("mybucket");]]>

    It is very important in a production setup to pass in at least two or three nodes of the cluster because if the first one in the list fails the other ones can be tried. After @@ -72,7 +86,8 @@ var bucket = ClusterHelper.GetBucket("mybucket");]]> with a list provided by the server (which contains all nodes of the cluster).

    As an alternative to programmatic configuration, the SDK also supports configuration via config file and soon will support JSON config files. Here is an example of an App.Config - that is equivalent to the configuration above: + that is equivalent to the configuration above and assumes that you are connecting to + Couchbase Server 5.0 or greater using RBAC: @@ -80,7 +95,7 @@ var bucket = ClusterHelper.GetBucket("mybucket");]]> - + @@ -98,8 +113,8 @@ var bucket = cluster.OpenBucket("mybucket");]]> ClusterHelper.Initialize("couchbaseClients/couchbase");

    More buckets can be open at the same time if needed:

    - +

    If more than one bucket is open at a time, the underlying internals of the client makes sure to utilize the resources as best as possible, that is sharing sockets, thread pools and so on.

    diff --git a/content/sdk/dotnet/n1ql-queries-with-sdk.dita b/content/sdk/dotnet/n1ql-queries-with-sdk.dita index e1cceb008f..128062026a 100644 --- a/content/sdk/dotnet/n1ql-queries-with-sdk.dita +++ b/content/sdk/dotnet/n1ql-queries-with-sdk.dita @@ -7,7 +7,7 @@ You can perform N1QL queries via the Couchbase .NET client. See Couchbase Developer documentation for a quick intro to N1QL.

    To issue N1QL queries, you should create a QueryRequest object, and pass @@ -152,5 +152,50 @@ foreach (var l in query) query to the (likely) thread pool in a non-blocking manner. Finally, we iterate through the rows returned by server in the QueryResult object.

    • + + + +
    Scan Consistency

    + Setting a staleness parameter for queries, with scan_consistency, enables a tradeoff + between latency and (eventual) consistency. A N1QL query using the default Not Bounded Scan Consistency will not wait for any + indexes to finish updating before running the query and returning results, meaning that results are returned quickly, but the + query will not return any documents yet to be indexed.

    + +

    + With Scan Consistency set to RequestPlus, all document changes and index updates are processed before + the query is run. Select this when consistency is always more important than performance. For a middle ground, + AtPlus is a "read your own write" (RYOW) option, which means it just waits for the new documents that + you specify to be indexed, rather than an entire index of multiple documents. + See the examples + for how to use AtPlus for the best performance balance for many circumstances. +

    +
    + + + +
    Streaming Large Result Sets

    By default the .NET SDK will fetch the entire result set from the server, + and de-serialize the entire set in-memory on the application. For smaller result sets this is + fine, however with large result sets this causes the memory footprint to grow linearly upwards. + This has an impact on CLR Garbage Collection and can lead to poor application performance, and + eventually an OutOfMemoryException may be thrown. To avoid this, it is + suggested that large results sets be streamed by setting the UseStreaming + property on the QueryRequest object:

    + +

    var request = new QueryRequest("SELECT * FROM `travel-sample`;").UseStreaming(true); +using (var result = _bucket.Query<DocumentContent>(request)) +{ + foreach (var doc in result) + { + Console.WriteLine(doc); + } +} + When streaming is enabled, the client will start a persistent connection with the server and + only read the header until the Rows are enumerated; then, each row or JSON object will be + de-serialized. The net effect is that the memory footprint of the application will stay a + constant and not increase linearly, and the Garbage Collector will collect objects created + during de-serialization in the first generation.

    +
    +     diff --git a/content/sdk/dotnet/sample-app-backend.dita b/content/sdk/dotnet/sample-app-backend.dita index c7af11ad88..34296c6efd 100644 --- a/content/sdk/dotnet/sample-app-backend.dita +++ b/content/sdk/dotnet/sample-app-backend.dita @@ -65,14 +65,8 @@ method bodies and try to take it from there.

    This tutorial focuses on querying through N1QL and FTS rather than - mapreduce views. If you want information about using views, see the following - resources:
      -
    • General Information about views: -
      • -
    • Querying views:
      • -
    + views. If you want information about using views, see + Views.
    Walking Through the API diff --git a/content/sdk/dotnet/scan-consistency-examples.dita b/content/sdk/dotnet/scan-consistency-examples.dita new file mode 100644 index 0000000000..bcbf7bc1a7 --- /dev/null +++ b/content/sdk/dotnet/scan-consistency-examples.dita @@ -0,0 +1,53 @@ + + + + Scan Consistency Using the .NET SDK with Couchbase Server + Using Scan Consistency + You can balance performance against consistency in N1QL queries via the Couchbase .NET client and the AtPlus option. + +

    + Setting a Scan Consistency, lets you choose between NotBounded (the default), for sharpest performance; RequestPlus, for strongest consistency; and AtPlus, for a good balance between increased performance and completeness of results. +

    + +

    In order to use at_plus, you'll need to make sure that you explicitly set UseEnhancedDurability to true when setting up your ClientConfiguration:config.BucketConfigs = new Dictionary<string, BucketConfiguration> { + { + "travel-sample", new BucketConfiguration + { + UseEnhancedDurability = true + } + } +};

    + +

    In the example below, the N1QL query will wait only for the new document to be indexed.private static void AtPlusExample() +{ + Console.WriteLine("========= AtPlus"); + + // get the current count + var getResult = _bucket.Query<dynamic>("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'") + .Rows.First(); + Console.WriteLine($"Initial count: {result1.airportCount}"); + + // insert a new airport + var doc = new Document<dynamic> + { + Id = "ScanConsistency::airport::" + _random.Next(10000), + Content = new + { + type = "airport" + } + }; + var insertResult = _bucket.Insert(doc); + + // get the count again, creating mutation state from insert result + var state = MutationState.From(insertResult.Document); + var request = new QueryRequest("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'") + .ConsistentWith(state); + var queryResult = _bucket.Query<dynamic>(t).Rows.First(); + Console.WriteLine($"Count after insert with AtPlus: {result2.airportCount}"); +}

    +
    + +     +     \ No newline at end of file diff --git a/content/sdk/dotnet/sdk-authentication-overview.dita b/content/sdk/dotnet/sdk-authentication-overview.dita index bb71782463..525eecda20 100644 --- a/content/sdk/dotnet/sdk-authentication-overview.dita +++ b/content/sdk/dotnet/sdk-authentication-overview.dita @@ -1,56 +1,56 @@ - + Authentication - + RBAC requires that users pass appropriate credentials to Couchbase Server in order to access cluster-resources. The SDK provides enhanced support for the passing of credentials. - + - +
    - -
    + +
    - +
    - + Passing Credentials - +

    - The most recent versions of the SDK are RBAC-enabled, and provide - enhanced support for authentication. Code examples for .NET can be found in + The most recent versions of the SDK are RBAC-enabled, and provide + enhanced support for authentication. Code examples for .NET can be found in Getting Started with the SDK, and also as part of the Sample Code provided to demonstrate user management.

    - +
    - +
    Upgrading to RBAC-Enabled SDK-Versions - +

    - The last pre-RBAC version of the .NET SDK was 2.4.2. The first - RBAC-enabled + The last pre-RBAC version of the .NET SDK was 2.4.2. The first + RBAC-enabled version was - 2.4.4. See the SDK + 2.4.4. See the SDK Release Notes for more information.

    - +

    With the most recent versions of the SDK, the legacy authentication interface and @@ -59,41 +59,40 @@ whose version is either 5.0 and beyond, or pre-5.0.

    - +
    - Further Information on RBAC + Further Information on RBAC - +

    All aspects of the Couchbase RBAC system are covered in the section Authorization. Specifically, for information on:

    - +
    • Adding Users and assigning roles, by means of - the Couchbase Web Console, see + the Couchbase Web Console, see Creating and Managing Users with the UI. - +

      - +

      • - +
    • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. - + Roles. +

      - +

      • - +
    • Resources controlled by Couchbase RBAC, see Resources @@ -101,6 +100,123 @@
    + +
    + + Certificate-Based Authentication + + +

    + Couchbase Server supports the use of X.509 certificates to authenticate clients (only available in + the Enterprise Edition, not the Community Edition). This allows + authenticated users to access specific resources by means of the data service (no other service + is currently available to clients through this form of authentication). +

    + +

    + The process relies on a certificate + authority, for the issuing of certificates that validate identities. A certificate includes + information such as the name of the entity it identifies, an expiration date, the name of the authority + that issued the certificate, and the digital signature of the authority. + A client attempting to access Couchbase Server can present a certificate to the server, allowing + the server to check the validity of the certificate. If the certificate is valid, the user + under whose identity the client is running, + and the roles assigned that user, are verified. If the assigned roles are appropriate for + the level of access requested to the specified resource, access is granted. +

    + +

    + Note that this means that the explicit authentication process otherwise required by Couchbase Role-Based + Access Control — whereby, in .NET, username and password are passed by means of the + authenticate method on the Cluster object — is not needed. + (See + Start Using the SDK for + an example of such standard authentication.) +

    + +

    + For a more detailed conceptual description of using certificates, see + Certificate-Based + Authentication. +

    + +

    + For sample procedures whereby certificates can be generated and deployed, see + X.509 for TLS. + Note that this page includes the steps whereby a Java keystore is created, to enable + certificate-based authentication by a Java client; for Windows there are a couple of more steps: + you’ll need to generate a Personal Information Exchange (.PFX) file and store that in the Windows + cert store. + +

    + +
    + +
    + + Authenticating a .NET Client by Certificate + + +

    + To authenticate with Couchbase Server by means of a client certificate, you will need + a key - although the key can be generated following the instructions on the + X.509 for TLS page, + you do not need a Java Key Store. +

    +

    Once you have generated the .pem file, for windows you'll need to use it to create a + .pfx: + openssl pkcs12 -export -in cert.cer -inkey key.pem -out certificate.pfx -certfile CA.cer +

    +

    The .pfx is then stored in the Windows certificate store of the application server by copying it to that server and + then right clicking on the file and selecting "Install Certificate" and following the steps (there are a couple + of ways to do this, but this is the simplest).

    +

    Once the .pfx is installed, the client configuration will have to be changed to use cert as opposed to the + usual Username and Password: + var config = new ClientConfiguration +{ + UseSsl = true +}; + +var cluster = new Cluster(config); +var authenticator = new CertAuthenticator( + new PathAndPasswordOptions + { + Path = "C:\path\to\client.pfx", + Password = "password" + } +); +cluster.Authenticate(authenticator); + +var bucket = cluster.OpenBucket();

    +
    + +
    + + Using a Different Certificate Factory + + +

    Note that there is also a way to query the cert store directly by using a different certificate factory: + var config = new ClientConfiguration +{ + UseSsl = true +}; + +var cluster = new Cluster(config); +var authenticator = new CertAuthenticator( + new CertificateStoreOptions + { + StoreLocation = StoreLocation.LocalMachine, + StoreName = StoreName.Root, + X509FindType = X509FindType.FindByIssuerName, + FindValue = "MyCompanyIntermediateCA" + } +); +cluster.Authenticate(authenticator); + +var bucket = cluster.OpenBucket();

    + +
    + - + diff --git a/content/sdk/dotnet/sdk-user-management-overview.dita b/content/sdk/dotnet/sdk-user-management-overview.dita index 736b3543dc..f80e70478a 100644 --- a/content/sdk/dotnet/sdk-user-management-overview.dita +++ b/content/sdk/dotnet/sdk-user-management-overview.dita @@ -66,8 +66,7 @@
  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    diff --git a/content/sdk/dotnet/start-using-sdk.dita b/content/sdk/dotnet/start-using-sdk.dita index 0c67cdb39e..9bec8870a5 100644 --- a/content/sdk/dotnet/start-using-sdk.dita +++ b/content/sdk/dotnet/start-using-sdk.dita @@ -230,7 +230,7 @@ var bucket = cluster.OpenBucket("bucketname");

    • API ReferenceThe API reference is generated for each release and can - be found here.
    Contributing

    Couchbase welcomes community contributions to the .NET SDK. The + + + Threshold Logging Tracing through the SDK + Threshold Logging + + + +

    +
    + +
    +
    + +
    +
    + + +
    + + Threshold Logging in .NET + + +

    + Here is the code to override the default values of the tracer: +var tracer = new ThresholdLoggingTracer +{ + Interval = 5000, // 5 seconds + SampleSize = 5, + KvThreshold = 500000, // 500 ms + ViewThreshold = 1000000, // 1 second + N1qlThreshold = 1000000, // 1 second + SearchThreshold = 1000000, // 1 second + AnalyticsThreshold = 1000000 // 1 second +}; +var config = new ClientConfiguration(); +config.Tracer = tracer; + +var cluster = new Cluster(config); +

    +
    + + diff --git a/content/sdk/dotnet/tracing-from-the-sdk.dita b/content/sdk/dotnet/tracing-from-the-sdk.dita new file mode 100644 index 0000000000..cd834e7a6f --- /dev/null +++ b/content/sdk/dotnet/tracing-from-the-sdk.dita @@ -0,0 +1,163 @@ + + + + Tracing from the .NET SDK with Couchbase Server + Tracing from the SDK + Use tracing from the SDK to discover timeouts and bottlenecks across the network and cluster. + +

    For our example, we will customize the threshold logging tracer settings in the ClientConfiguration. By default it + will log every minute (if something is found) and only sample the top 10 slowest operations. The default threshold for key/value operation is 500 milliseconds. We shall + set them so low that almost everything gets logged - not something you should do in production!var tracer = new ThresholdLoggingTracer + { + KvThreshold = 1, // 1 microsecond + SampleSize = int.MaxValue, + Interval = 1000 // 1 second + }; + This will set our threshold for key/value operations to one microsecond, log the found operations every second, and set the sample size to a very large value so everything will be logged.

    + +

    With these configs in place we are ready to run some operations. Below, we read some documentss from the travel-sample bucket and, if found, write them back with an upsert - giving us both read and write operations to log.// Connect + var config = new ClientConfiguration(); + config.Tracer = tracer; + + var cluster = new Cluster(); // connects to cluster using localhost + cluster.Authenticate("username", "password"); + + // Load a couple of docs and write them back + var bucket = cluster.OpenBucket("travel-sample"); + for (var i = 0; i < 5; i++) + { + var result = bucket.Get<dynamic>("airline_1" + i); + if (result.Success) + { + bucket.Upsert(result.Id, result.Value); + } + } + + Thread.Sleep(TimeSpan.FromMinutes(1));

    + +

    Run the code, and you will see something like the following in the logs:27 11:51:09,129 [4] INFO Couchbase.Tracing.ThresholdLoggingTracer - Operations that exceeded service threshold: +[ + { + "service":"kv", + "count":5, + "top":[ + { + "operation_name":"Set", + "last_operaion_id":"0x9", + "last_local_address":"10.211.55.3:58679", + "last_remote_address":"10.112.181.101:11210", + "last_local_id":"8fe25176f5cb5068/7a424b9f2ab5e6a8", + "last_dispatch_us":31153, + "total_us":39299, + "encode_us":6791, + "dispatch_us":31153, + "server_us":17290, + "decode_us":909 + }, + { + "operation_name":"Get", + "last_operaion_id":"0xa", + "last_local_address":"10.211.55.3:58678", + "last_remote_address":"10.112.181.101:11210", + "last_local_id":"8fe25176f5cb5068/1ca98582755c6f19", + "last_dispatch_us":18205, + "total_us":23802, + "encode_us":7, + "dispatch_us":18205, + "server_us":280, + "decode_us":1653 + }, + { + "operation_name":"Get", + "last_operaion_id":"0xb", + "last_local_address":"10.211.55.3:58679", + "last_remote_address":"10.112.181.101:11210", + "last_local_id":"8fe25176f5cb5068/7a424b9f2ab5e6a8", + "last_dispatch_us":1657, + "total_us":1830, + "encode_us":3, + "dispatch_us":1657, + "server_us":135, + "decode_us":29 + }, + { + "operation_name":"Get", + "last_operaion_id":"0xc", + "last_local_address":"10.211.55.3:58678", + "last_remote_address":"10.112.181.101:11210", + "last_local_id":"8fe25176f5cb5068/1ca98582755c6f19", + "last_dispatch_us":1373, + "total_us":1481, + "encode_us":2, + "dispatch_us":1373, + "server_us":22, + "decode_us":12 + }, + { + "operation_name":"Get", + "last_operaion_id":"0xd", + "last_local_address":"10.211.55.3:58679", + "last_remote_address":"10.112.181.101:11210", + "last_local_id":"8fe25176f5cb5068/7a424b9f2ab5e6a8", + "last_dispatch_us":4876, + "total_us":5086, + "encode_us":8, + "dispatch_us":4876, + "server_us":14, + "decode_us":8 + } + ] + } +]For each service (only kv-based on this workload), the threshold log reporter will show you the total number of recorded ops (through count), and give you the top slowest ops sorted by their latency. Since only airline_10 exists in the bucket you will see five document fetches but only one mutation.

    + + +
    + + Output fields in detail. + +

    Let's highlight a single operation, and explain each field in a little more detail:{ + "operation_name":"Get", + "last_operaion_id":"0xc", + "last_local_address":"10.211.55.3:58678", + "last_remote_address":"10.112.181.101:11210", + "last_local_id":"8fe25176f5cb5068/1ca98582755c6f19", + "last_dispatch_us":1373, + "total_us":1481, + "dispatch_us":1373, + "server_us":22, + "decode_us":12 +}

    +

    This tells us the following:

      +
    • operation_name: The operation type, eg for KV it is the command type 'Get'.
      • +
    • last_operation_id: The last unique ID for the opeation (in this case the opaque value), useful for diagnosing and troubleshooting in combination with the last_local_id.
      • +
    • last_local_address: The local socket used for this operation.
      • +
    • last_remote_address: The remote socket on the server used for this operation. Useful to figure out which node is affected.
      • +
    • last_local_id: With Server 5.5 and later, this id is negotiated with the server and can be used to correlate logging information on both sides in a simpler fashion.
      • +
    • last_dispatch_us: The time when the client sent the request and got the response took around 1 millisecond.
      • +
    • total_us: The total time it took to perform the full operation: here around 1.5 milliseconds.
      • +
    • dispatch_us: The amount of time observed between sending the request over the network to when a response was received.
      • +
    • server_us: The server reported that its work performed took 22 microseconds (this does not include network time or time in the buffer before picked up at the cluster).
      • +
    • decode_us: Decoding the response took the client 12 microseconds.

    +

    You can see that if the thresholds are set the right way based on production requirements, without much effort slow operations can be logged and pinpointed more easily than before.

    +
    + + +
    + + Timeout Visibility. + +

    Previously, when an operation takes longer than the timeout specified allows, a TimeoutException is thrown. It usually looks like this:.2018-06-27 12:26:13,755 [Worker#STA_NP] WARN Couchbase.IO.ConnectionBase - Couchbase.IO.SendTimeoutExpiredException: The operation has timed out. {"s":"kv","i":"0x1","c":"8d063f1e0b70ebb8/cd65184a378ae5fc","b":"travel-sample","l":"10.211.55.3:58754","r":"10.112.181.101:11210","t":5000} + at Couchbase.IO.MultiplexingConnection.Send(Byte[] request) + at Couchbase.IO.Services.IOServiceBase.Execute[T](IOperation`1 operation, IConnection connection) + at Couchbase.IO.Services.IOServiceBase.EnableServerFeatures(IConnection connection) + at Couchbase.IO.Services.IOServiceBase.CheckEnabledServerFeatures(IConnection connection) + at Couchbase.IO.Services.PooledIOService..ctor(IConnectionPool connectionPool) + at Couchbase.IO.Services.SharedPooledIOService..ctor(IConnectionPool connectionPool) + at Couchbase.IO.IOServiceFactory.<>c__DisplayClass0_0.<GetFactory>b__0(IConnectionPool pool) + at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)Now the timeout itself provides you valuable information like the local and remote sockets, and the operation id, as well as the timeout set and the local ID used for troubleshooting. You can take this information and correlate it to the top slow operations in the threshold log.

    + +

    The TimeoutException now provides you more information into what went wrong and then you can go look at the log to figure out why it was slow. +

    +
    + +     diff --git a/content/sdk/durability.dita b/content/sdk/durability.dita index 1c8d522aff..4e592f549b 100644 --- a/content/sdk/durability.dita +++ b/content/sdk/durability.dita @@ -50,7 +50,7 @@ successfully stored to disk and/or replicated to replica nodes. Upon receiving such a failure response, the application may retry the operation or mark it as a failure. In any event, the application will never be under the mistaken assumption that the - modification was performed when it fact it was lost inside one of the aforementioned + modification was performed when in fact it was lost inside one of the aforementioned queues.

    Configuring durability

    Durability requirements may be present as diff --git a/content/sdk/encryption.dita b/content/sdk/encryption.dita new file mode 100644 index 0000000000..053d774cbf --- /dev/null +++ b/content/sdk/encryption.dita @@ -0,0 +1,158 @@ + + + + Field Level Encryption + Fields within a document can be securely encrypted by the SDK, to support FIPS-140-2 compliance. + + + + + + +

    + Overview + + Actors & Responsibilities in Field Level Encryption + + +

    This is a client-side implementation, with key management and initialization of data encryption done during configuration of the SDK, which then exposes the API during runtime, for normal read/write operations.

    +
    + +
    + Algorithm and Key Store +

    In Couchbase Data Platform 5.5, AES-256, AES-128, and RSA are all supported. Native Keystores (including Java Key Store and Windows Certificate Store) are supported, as well as an in-memory keystore for development and testing. More options for key stores and encryption algorithms may appear in future SDK releases.

    +

    Here is an example of an in-memory store for development and testing called the “InsecureKeyStore”, to reflect its lack of security:public class InsecureKeyProvider : IKeyProvider +{ + private readonly Dictionary<string, string> _keys = new Dictionary<string, string>(); + + public string GetKey(string keyname) + { + return _keys[keyname]; + } + + public void StoreKey(string keyname, string key) + { + _keys[keyname] = key; + } +}The keys are stored by name in a dictionary and retrieved using the same name. See the sample code page for more secure code.

    +
    + +
    + Field Encryption Format +

    Behind the API, the following format is used internally to encompass both the temporary field name used to hold the encrypted value, plus the additional metadata associated with the algorithm used and the public key:__[PREFIX]_[FIELDNAME]” : { + “kid” : “[KEY_IDENTIFIER]”, + “alg”: “[ALGORITHM”], + “ciphertext”: “[BASE64_ENCRYPTED_DATA]”, + “sig”: “[BASE64_HMAC_SIGNATURE]”, + “Iv” : [“INITIALIZATION_VECTOR”] +}This is how Couchbase stores the JSON internally, but it is exposed to the developer as a public API (which currently has an uncommited status). For your preferred SDK, an API reference is available, linked from as well as sample code.

    +
    + +
    + Field Types +

    The encryption payload can be any object that comes after a key in a JSON document.

    + + Supported Field Types + + + + + + + Type + Example + Payload + + + + + string + "magicWord":"xyzzy" + "xyzzy" + + + object + "score":{"dance":10,"presentation":3} + {"dance":10,"presentation":3} + + + numeric + "myint":10 + 10 + + + null + "isnull":null + null + + + +
    +
    + +
    + Auditing +

    For auditing purposes, the encrypted fields must be discoverable, for example via N1QL.

    +
    + +
    + Data Safeguards +

    Under error conditions, to prevent data loss, an error in decryption or encrytpion in any part of an operation will cause the whole operation to fail with an exception, to prevent unnoticed loss of data.

    +

    It's important that encrypted fields be treated as “special” fields and not mutated by APIs other than through the + Field Level Encryption (FLE) API. For example, for AES-HMAC-SHA256 the entire temporary field is signed; if any changes are made to any fields + (“alg”, “kid”, “ciphertext”, “sig” or “iv”) then the decryption will fail because the signature has changed.

    +
    + + +
    + Packaging +

    Field Level Encryption for all SDKs is a separate package from the Couchbase SDK itself; the APIs are extensions of + the SDK, but the SDK does not have a dependency on FLE. This means you can install the relevant SDK without being + dependant upon a suite of crypto libraries.

    +
    + + +
    + Best Practices + +

    + You are strongly recommended: +

    + +
      + +
    • + Always to maintain the secrecy of your encryption key: by means of either a secure key ring, or a + hardware security module (HSM). + +

      + +

      + +
      • + +
    • + Always to use a standard encryption library to encrypt data; and never to rely on + in-house encryption or key-management solutions. + +

      + +

      + +
      • + +
    • + Never, when managing documents, to include sensitive information in the document ID. + +

      + +

      + +
      • + +
    + +
    + + + diff --git a/content/sdk/full-text-search-overview.dita b/content/sdk/full-text-search-overview.dita index 8301372633..e42d2013b8 100644 --- a/content/sdk/full-text-search-overview.dita +++ b/content/sdk/full-text-search-overview.dita @@ -1,305 +1,136 @@ - Full Text Search - Couchbase Server Full Text Search (FTS) enables you to create, manage, and query full - text indexes on JSON documents stored in a Couchbase bucket. You should use search instead - of query when your application requires natural language processing when - searching. + + + Full Text Search + + + + Full Text Search (FTS) lets you create, manage, and query specially purposed indexes, + defined on JSON documents within a Couchbase bucket. + + + -
    - What is Full Text Search -

    Couchbase FTS is similar in purpose to other search software such as - ElasticSearch or Solr. Couchbase FTS is not intended as a - replacement for third party search software if search is at the core of your - application. It is a simple and lightweight way to add search to your Couchbase data - without deploying additional software and servers. If you have many queries which - look like SELECT ... field1 LIKE %pattern% OR field2 LIKE %pattern, - then full-text search may be right for you.

    -
    -
    Executing Your First Search

    Our first search query will be done - against the travel-sample bucket (install it, if you haven't done - so already).

      -
    1. Go to the Couchbase Web Console (for example, http://localhost:8091) and log - in with your administrative username and password.
      2. -
    2. Select - Indexes - Full Text - (at the top bars).
      3. -
    3. In the dropdown menu on the top left (-- choose full text index or - alias --) select travel-search.
      4. -
    4. In the input box (on the right), type cheese and - click Search.
      5. -
    You will see a list of document IDs that contain the given search term. You can - click on any document ID to see the full document. You can also click on - Advanced to see the raw query as it is sent over the REST - API. If you click on the command-line curl example checkbox, - you can simply copy/paste the output to your terminal and execute the search - again.

    Here's a similar version of the search done using curl - on the command - line:curl -XPOST -H "Content-Type: application/json" \ - http://localhost:8094/api/index/travel-search/query \ - -d '{"query":{"query":"cheese"}, "size": 1}' | json_pp

    (Note - that json_pp is just a JSON formatter. You can omit the pipe if you - do not have it - installed).{ - ... - "hits" : [ - { - "index" : "travel-search_2c8e9e3d5a3638a8_b7ff6b68", - "score" : 1.20476244729454, - "locations" : { - "name" : { - "cheese" : [ - { - "end" : 12, - "array_positions" : null, - "start" : 6, - "pos" : 2 - } - ] - }, - ... - }, - "id" : "landmark_27808" - } - ] -}(Most - of the output has been redacted)

    The same query using the Python - SDK:from couchbase.bucket import Bucket -import couchbase.fulltext as FT - -cb = Bucket() -results = cb.search('travel-search', FT.StringQuery('cheese'), limit=5) -for hit in results: - print('Found in document ID={}. Score={}'.format(hit['id'], hit['score']))Found in document ID=landmark_27808. Score=1.20476244729 -Found in document ID=landmark_8689. Score=0.961675281363 -Found in document ID=landmark_1154. Score=0.883110932061 -Found in document ID=landmark_1163. Score=0.846040674514 -Found in document ID=landmark_15133. Score=0.81847489864

    The - result contains one or more hits, where each hit contains information - about the location of the match within a document. This includes the relevance - score as well as the location where the match was - found.

    -
    - Making Your Bucket Searchable -

    In order to execute a search query you must first define a search index. You can - define a search index by using the Couchbase Web Console.

      -
    1. Go to the Couchbase Web Console (for example, http://localhost:8091) and log - in with your administrative username and password.
      2. -
    2. Select - Indexes - Full Text - New Full Text Index -
      3. -
    3. Type the desired name of the search index in the Index - Name field.
      4. -
    4. Select the bucket you would like to associate the search with in the - Bucket field
      5. -
    The default settings are sufficient for most cases. You can edit the index - later, specifying how certain fields may be analyzed and which fields to index.

    -

    Once you've created your index, your can query it by using the methods above, - replacing travel-search with the name you used to create the - index.

    -

    To learn more about making your buckets searchable, see the - section of the - full text search documentation.

    -
    -
    Query Types

    The query executed in the previous section is called a - string query. This type of query searches for terms based on a special - type of input string. The query string +description:cheese - -country:france will match documents which contain cheese in - their description field, but are not located in France. - String queries are ideal for searchbox fields to allow users to provide more - specialized query criteria. You can read more about the Query String syntax - in the full text search documentation.

    -

    There are many other query types available. These query types differ - primarily in how they interpret the search term: whether it is treated as a phrase, - a word, an exact match, or a prefix: A Match query searches for the - input text within documents and is the simplest of queries. Match - Phrase query searches for documents in which a specific phrase - (i.e. one or more terms, such as "french cheese tasting") is - present. A Prefix query searches documents which contain terms - beginning with the supplied prefix.

    There are some other specialized queries, - such as Wildcard and Regexp queries which allow you to use - wildcards (Couch?base') or regular expressions (Couchbase - (php|python) SDK).

    Below are two code snippets showing how the query - for ch is treated differently when using a Prefix Query - versus a Match - Query.

    results = cb.search('travel-search', FT.MatchQuery('ch'), limit=5) -for r in results: - print(' Result: ID', r['id']) - for location, terms in r['locations'].items(): - print(' ', location, terms.keys()) Result: ID airline_1442 - iata dict_keys(['ch']) - Result: ID landmark_35848 - image_direct_url dict_keys(['ch'])results = cb.search('travel-search', FT.PrefixQuery('ch'), limit=5) -for r in results: - print(' Result: ID', r['id']) - for location, terms in r['locations'].items(): - print(' ', location, terms.keys()) Result: ID hotel_15912 - reviews.content dict_keys(['check', 'cheese', 'charge', 'checkout', 'chairs', 'chances', 'choice', 'checked', 'cheapcaribbean', 'cheeses', 'charged', "church's", 'cheaper', 'chicken', 'change']) - reviews.author dict_keys(['christiansen']) - Result: ID hotel_33886 - reviews.content dict_keys(['check', 'chose', 'chips', 'choosing', 'chairs', 'channels', 'changed', 'choice', 'checked', 'chair', 'chocolate', 'chaise', 'checking', 'chicken', 'change', 'choose', 'charter', 'cheerful']) - reviews.author dict_keys(['christy']) - Result: ID hotel_16634 - reviews.content dict_keys(['check', 'chocolate', 'chinese', 'chairs', 'children', 'chips', 'chilis', 'chilly', 'checked', 'choice', 'chicago', 'childrens', "church's", 'cheaper', 'chicken', 'choose', 'christina', 'choices']) - Result: ID hotel_37318 - reviews.content dict_keys(['check', 'choices', 'chairs', 'children', 'changed', 'choice', 'charge', 'challenging', 'chair', 'childrens', 'chicken', 'change', 'choose', 'chambermaid', 'chichen', 'child']) - city dict_keys(['cheshire']) - Result: ID hotel_21723 - reviews.content dict_keys(['check', 'chairs', 'cheesy', 'changed', 'checked', 'chair', 'charging', 'chaotic', 'charge', 'chapel', 'change', 'choose', 'children', 'cheep', 'chef', 'child']) - content dict_keys(['cheapie']) - public_likes dict_keys(['christop'])As - can be seen in the above examples, the Term assumes the search input is - an actual term to search for (ch) and therefore rejects things such - as chose, chairs and similar.

    -
    -
    - Compound Queries -

    You can compose queries made of other queries. You can use a Conjunction - or Disjunction query which contains one or more queries that the document - should match (a Disjunction query can be configured with the number of - required subqueries that must be matched). You may also use a Boolean - query that itself contains sub queries which should, must, and must - not be matched.

    -

    Compound queries can be used to execute searches such as find any landmark - containing "cheese" and also containing one of "wine" , "crackers" , or "old", - but does not contain "lake" or - "ocean":results = cb.search('travel-search', - FT.BooleanQuery( - must=FT.TermQuery('cheese'), - should=[FT.TermQuery('wine'), FT.TermQuery('crackers')], - must_not=[FT.TermQuery('lake'), FT.TermQuery('ocean')]), - limit=5) - -for r in results: - print('ID', r['id']) - for location, terms in r['locations'].items(): - print('\t{}: {}'.format(location, terms.keys()))ID landmark_25779 - content: dict_keys(['cheese', 'crackers']) -ID landmark_7063 - content: dict_keys(['wine']) - alt: dict_keys(['cheese']) - name: dict_keys(['wine']) -ID landmark_16693 - content: dict_keys(['cheese', 'wine']) -ID landmark_27793 - content: dict_keys(['cheese', 'wine']) -ID landmark_40690 - content: dict_keys(['cheese', 'wine'])

    -

    When using compoound queries, you can modify any subquery's - boost setting to increase its relevance and scoring over - other subqueries, affecting the ordering.

    -
    -
    - Other Query Types -

    There are other query types you can use, such as Date Range and - Numeric Range queries which match documents matching a certain time - span or value range. There are also debugging queries such as Term and - Phrase queries which perform exact queries (without any - analysis).

    -

    For a quick overview of all the available query types, see the - section of the full text search documentation.

    -
    -
    - Query Options -

    You can specify query options to modify how the search term is treated. This section - will enumerate some common query options and how they affect query results.

      -
    • field: This option restricts searches to a given field. - By default searches will be executed against all fields.
      • -
    • fuzziness: Sets the leniency of the matching - algorithm. A higher fuzziness value may result in less relevant matches - being considered
      • -
    • analyzer: Sets the analyzer to be used for the search - term.
      • -
    • limit: Limits the number of search results to be - returned.
      • -
    • skip: Start returning after this many results. This may - be used in conjunction with limit to use - pagination.
      • -

    -
    -
    - Search Results -

    After you have executed a search, you will be given a set of results, containing - information about documents which match the query. In the raw JSON payload, the - server returns an object with a hits property, which contains a - search result.

    -

    The search result itself is a JSON object containing:

      -
    • id: The document ID of the hit
      • -
    • score: How relevant the result is to the initial search - query. Search results are always ordered by score, with highest-scored hits - appearing first.
      • -
    • locations: A JSON object containing information about - each match in the document. Its keys are document paths (in the - N1QL sense) - where matches may be found, and its values are arrays - that contain the match location. The match location is a JSON - object whose keys are the matched terms found, and whose values are - locations:
        -
      • start: The character offset at which the - matched text begins
        • -
      • end: The character offset at which the matched - text ends
        • -
      • pos: The word-position of the matched result. - This indicates how far deep the match is, in respect to words. For - example if the searched term was schema, and the - matched text was: Ahout NoSQL schema organization, - the pos would be 3, or the - third word in the field.
        • -
      • -

    -

    To learn more about the response format used by the FTS service, see the - section of the full text search documentation. - Couchbase SDKs may abstract some of the fields or provide wrapper methods around them.

    -
    -
    - Aggregation and Statistics (Facets) -

    You may perform search result aggregation and statistics using facets. Facets - allow you to specify aggregation parameters in your query. When the query results - are received, aggregation results are returned alongside the actual query hits.

    -

    You can use a Term Facet to count the number of times a specific term appears - in the - resultsresults = cb.search('beer-search', FT.MatchQuery('hops'), limit=5, - facets={'terms': FT.TermFacet('description', limit=5)}) -for result in results: - # handle results - pass - -pprint(results.facets['terms']){'field': 'description', - 'missing': 9, - 'other': 30725, - 'terms': [{'count': 782, 'term': 'hops'}, - {'count': 432, 'term': 'beer'}, - {'count': 365, 'term': 'ale'}, - {'count': 327, 'term': 'malt'}, - {'count': 130, 'term': 'hop'}], - 'total': 32761}

    -

    You can likewise use a Date Range Facet to count the number of results by - their age, and Numeric Range Facet to count results using an arbitrary - numeric range.

    -

    To learn more about facets, see the section of the - full text search documentation.

    -
    -
    - Partial Search Results -

    The FTS service splits the indexing data between several pindexes. Because of that, - you may encounter situations where only a subset of the pindexes could provide results - (eg. if some pindex nodes are not online...).

    -

    What happens in this case is that FTS returns a list of partial results, and - notifies you that one or several errors also happened. You can inspect the - errors, which will each correspond to an failing pindex, via the SDK. Of course, the - partial results (the result from healthy pindexes) are still available through the - usual methods in the SDK result representation.

    -
    +
    + + + What is Full Text Search? + + +

    + Full Text Search provides extensive capabilities for natural-language querying: + this allows special search-constraints to be applied to text-queries. Results can + be scored, to indicate match-relevancy; and result-sets ordered + correspondingly. Conjunctive and disjunctive searches can be performed, + whereby common result-subsets from multiple queries can either be returned or + omitted. +

    + +

    + A full overview of Full Text Search is provided in + Full Text Search: Fundamentals. + This includes information on the principal features of Couchbase Full Text Search, its + architecture, and the latest feature-additions. Other information-sources include: +

    + +
      +
    • + Performing Searches: An + explanation of the steps required to prepare for and perform Full Text Search. + +

      + +

      +
      • + +
    • + Searching from the UI: A + brief introduction to the Full Text Search user interface provided by the Couchbase Web Console, with a + step-by-step example of how to create a simple Full Text Index, and perform a search on it. + +

      + +

      +
      • + +
    • + Searching with the REST API: Basic + examples of how Full Text Search is performed with REST, and pointers to more complex examples. + +

      + +

      +
      • + +
    • + Creating Indexes: A + full description of the index-creation facility provided by the Couchbase Web Console, with + explanations of each component to be used, and illustrations of how indexes can be designed + to include specific subsets of documents and their fields. + +

      + +

      +
      • + +
    • + Understanding Analyzers: An + explanation of analyzers, which are used to process the text to be included in Full Text Indexes. + +

      + +

      +
      • + +
    • + Queries: A detailed + account of available query types, response objects, and result-sorting options. + +

      + +

      +
      • + +
    + +
    + +
    + + + Performing Full Text Search from the SDK + + +

    + Each language supported by the Couchbase SDK provides an API for the support of + Full Text Search querying. For detailed information, see the page for your language + of choice. +

    + +

    + A detailed example of performing Full Text Search queries from the Java SDK is + provided in + Searching from the SDK. +

    + +

    + Note that to access Full Text Search, users require appropriate roles. + The role FTS Admin + must therefore be assigned to those who intend to create indexes; and the role FTS Searcher to those who intend to + perform searches. For information on creating users and assigning roles, see + Authorization. +

    + + +
    + +     diff --git a/content/sdk/go.ditamap b/content/sdk/go.ditamap index e65d7ddcfc..1e70c894f9 100644 --- a/content/sdk/go.ditamap +++ b/content/sdk/go.ditamap @@ -18,6 +18,9 @@ + + + @@ -38,6 +41,7 @@ + @@ -49,5 +53,7 @@ + + diff --git a/content/sdk/go/collecting-information-and-logging.dita b/content/sdk/go/collecting-information-and-logging.dita index 2bd06d8d63..9ba327a69b 100644 --- a/content/sdk/go/collecting-information-and-logging.dita +++ b/content/sdk/go/collecting-information-and-logging.dita @@ -1,31 +1,45 @@ - + Collecting Information and Logging in the Go SDK with Couchbase Server Collecting Information

    The Go SDK offers simple logging of library internals to help debug issues. Logging may be configured on a global library-level basis. Note that the logging API is subject to change.

    -

    You can configure logging using the gocbcore.SetLogger, which accepts - an implementation of gocbcore.Logger. The SDK comes with two built-in +

    You can configure logging using the gocb.SetLogger, which accepts + an implementation of gocb.Logger. The SDK comes with two built-in Logger implementations, which can be instantiated using the - following methods:

      -
    • gocbcore.DefaultStdioLogger() returns a logger that logs + following methods:
        +
      • gocb.DefaultStdioLogger() returns a logger that logs errors and warnings. This is fairly non-disruptive and does not produce a lot of - output
        • -
      • gocbcore.VerboseStdioLogger() returns a logger that logs more + output.
        • +
      • gocb.VerboseStdioLogger() returns a logger that logs more detailed tracing information. This logger should only be used when trying to diagnose an issue.

      import ( - "github.com/couchbaselabs/gocb/gocbcore" + "github.com/couchbase/gocb" ) func main() { - gocbcore.SetLogger(gocbcore.DefaultStdioLogger()) + gocb.SetLogger(gocb.DefaultStdioLogger()) // Use the gocb library. } +

      It is also possible to provide other logger implementations to gocb.SetLogger. + Implementations must satisify the gocb.Logger interface.

      + + type Logger interface { + // Outputs logging information: + // level is the verbosity level + // offset is the position within the calling stack from which the message + // originated. This is useful for contextual loggers which retrieve file/line + // information. + Log(level LogLevel, offset int, format string, v ...interface{}) error +} +

      The gocb.DefaultStdioLogger() and gocb.VerboseStdioLogger() wrap + their gocbcore counterparts to provide a stable interface. The gocb versions + should be used.

      diff --git a/content/sdk/go/compatibility-versions-features.dita b/content/sdk/go/compatibility-versions-features.dita index 99abdde220..d75f4abc49 100644 --- a/content/sdk/go/compatibility-versions-features.dita +++ b/content/sdk/go/compatibility-versions-features.dita @@ -14,7 +14,7 @@
    • ✖ Unsupported: This combination is not tested, and is not within the scope of technical support if you have purchased a support agreement.
    • ◎ Compatible: This combination has been tested previously, and should be - compatible. This combination is not supported or recommended by our technical support + compatible. This combination is not recommended by our technical support organization. It is best to upgrade either the SDK or the Couchbase version you are using.
    • ✔ Supported:This combination is subject to ongoing quality assurance, and is @@ -27,69 +27,29 @@ - CB/SDK SDK 1.0 - SDK 1.1 - SDK 1.2 + SDK 1.1 & 1.2 SDK 1.3 - - CB 2.5 - ◎ - ◎ - ◎ - ◎ - - - CB 3.0 - ◎ - ◎ - ◎ - ◎ - CB 3.1 ◎ ✔ ✔ - ✔ - - - CB 4.0 - ✔ - ✔ - ✔ - ✔ - - - CB 4.1 - ✔ - ✔ - ✔ - ✔ - CB 4.5 - ✔ - ✔ - ✔ - ✔ - - - CB 4.6 - ✔ + CB 4.0 - 4.6 ✔ ✔ ✔ - CB 5.0 - ✔ + CB 5.0 - 5.1 ✔ ✔ ✔ diff --git a/content/sdk/go/encrypting-using-sdk.dita b/content/sdk/go/encrypting-using-sdk.dita new file mode 100644 index 0000000000..e0c8b12601 --- /dev/null +++ b/content/sdk/go/encrypting-using-sdk.dita @@ -0,0 +1,67 @@ + + + + Field Level Encryption from the Go SDK + Field Level Encryption is available in Couchbase Data Platform 5.5. + +
      + Packaging +

      The Couchbase Go SDK uses the gocbencryption library to provide functionality for encryption and decryption.go get github.com/couchbase/gocbencryption

      + +

      The Couchbase Go Field Level Encryption (FLE) library uses struct tags to specify which field(s) to apply encryption to and which algorithm to use. The struct tag key is cbcrypt and the value is of the form "provider". Here’s an example struct definition:type PersonAddress struct { + HouseName string `json:"houseName"` + StreetName string `json:"streetName"` +} + +type Person struct { + FirstName string `json:"firstName"` + LastName string `json:"lastName"` + Password string `json:"password" cbcrypt:"myAESProvider"` + Address PersonAddress `json:"address" cbcrypt:"myAESProvider"` +}

      +
      + + +
      + Configuration +

      You need to create a Key Store, a Provider, and a Transcoder. The provider is used to perform encyption/decryption and the transcoder is responsible for using the provider during operations. You can register multiple (uniquely aliased) providers with a transcoder. After installing the dependency you need to set up your Key Store, Provider, and Transcoder (note that the alias name of the provider matches the struct tags):keyStore := &gocbfieldcrypt.InsecureKeystore{ + Keys: map[string][]byte{ + "mypublickey": []byte("!mysecretkey#9^5usdk39d&dlf)03sL"), + "myhmackey": []byte("myauthpassword"), + }, +} + +aesProvider := &gocbfieldcrypt.AesCryptoProvider{ + Alias: "myAESProvider", + KeyStore: keyStore, + Key: "mypublickey", + HmacKey: "myhmackey", +} + +coder := gocbfieldcrypt.Transcoder{} +coder.Register("myAESProvider", aesProvider)

      + +

      Next you need to create a configuration to connect to your cluster and set your transcoder on the bucket:cluster, _ := gocb.Connect("…") +cluster.Authenticate(…) +bucket, _ := cluster.openBucket("…", "") +bucket.SetTranscoder(&coder)

      +
      + +
      + Transparent operation +

      You can then perform KV operations as usual and your data will be encrypted/decrypted automatically:person := Person{ + FirstName: "Barry", + LastName: "Sheen", + Password: "bang!", + Address: PersonAddress{ + HouseName: "my house", + StreetName: "my street", + }, +} + +bucket.Upsert("p1", person, 0)

      +
      + +       +       diff --git a/content/sdk/go/failure-considerations.dita b/content/sdk/go/failure-considerations.dita index a0b5bc59ee..b3f8c8f61f 100644 --- a/content/sdk/go/failure-considerations.dita +++ b/content/sdk/go/failure-considerations.dita @@ -69,8 +69,15 @@ cas, err := bucket.Get("key", &value) if err != nil { cas, err = bucket.GetReplica("key", &value, 0) } - - +

      + If the third parameter to GetReplica is greater than zero then only + that specific replica will be queried. Otherwise, a request will be sent to all replicas. + Once the SDK receives a successful response from a replica then it will attempt to cancel + all other requests and, once all requests are cancelled, assign the value from the successful + response. If no successful response is received then an ErrNoReplicas + error will be returned. As the value is assigned from the first successful response this means + that only one value will be assigned to the value pointer. +

    diff --git a/content/sdk/go/full-text-searching-with-sdk.dita b/content/sdk/go/full-text-searching-with-sdk.dita index 2923e87b20..7cfb82e5af 100644 --- a/content/sdk/go/full-text-searching-with-sdk.dita +++ b/content/sdk/go/full-text-searching-with-sdk.dita @@ -9,9 +9,16 @@ certain words or phrases. In the Go SDK you can search full-text indexes by using the SearchQuery object and executing it with the Bucket.ExecuteSearchQuery() API.

    -

    The following example shows how to send a simple Search query:

    -

    - query := gocb.NewSearchQuery("travel-search", cbft.NewTermQuery("office")) +

    The following example shows how to send a simple Search query (note that in this example + the cbft package also has to be imported independently): +import ( + "github.com/couchbase/gocb" + "github.com/couchbase/gocb/cbft" + "fmt" +) + +// ... +query := gocb.NewSearchQuery("travel-search", cbft.NewMatchQuery("office")) res, _ := bucket.ExecuteSearchQuery(query) for _, hit := range res.Hits() { fmt.Printf("%s\n", hit.Id) @@ -47,10 +54,405 @@ fmt.Printf("Total Hits: %d", res.Status().Total)

    AddFacet method. The AddFacet method accepts a facet name as well as a facet definiton You can create facets by instantiating a Facet object found in the gocb.cbft package.

    - query := gocb.NewSearchQuery("travel-search", cbft.NewTermQuery("office")) + query := gocb.NewSearchQuery("travel-search", cbft.NewMatchQuery("office")) query.AddFacet("countries", cbft.NewTermFacet("country", 5)) res, _ := bucket.ExecuteSearchQuery(query) fmt.Printf("Total Countries: %d", res.Facets()["countries"].Total)
    + + +
    Detailed Examples + +

    + The code example below demonstrates the Go SDK Full Text Search + API. The example assumes that Couchbase Server is running, and that + the username Administrator and + the password password provide authorization for performing the searches. + It also assumes that the travel-sample bucket + has been installed. + For information on creating users and managing roles, see + Authorization. + For information on installing sample buckets, see + Settings. +

    + +

    + The example also assumes the existence of three specific Full Text Indexes, defined on the + travel-sample bucket. These are: +

    + +
      +
    • + travel-sample-index-unstored: Uses only the default + settings. +
      • + +
    • + travel-sample-index-stored: Uses default settings, with one exception: + dynamic fields are stored, for the whole index. +
      • + +
    • + travel-sample-index-hotel-description: Indexes only the description + fields of hotel + documents, and disables the default type mapping. The index + has a custom analyzer named myUnicodeAnalyzer defined on it: the analyzer's + main characteristic is that it uses the unicode tokenizer. +
      • +
    + +

    + See + Creating Indexes + for details on how to create these indexes: they can be created interactively, by + means of the Couchbase Web Console; however, there may be greater efficiency + in using the Couchbase REST + API, as described + in the section + Index Creation with the REST API. + The JSON objects that constitute index-definitions (for + inclusion as bodies to the index-creation REST calls), are provided in + Demonstration Indexes. +

    + +

    + The example features the following Full Text Searches on the + travel-sample bucket, within Couchbase Server: +

    + +
      + +
    • + Simple Text Query on a single word, targeting an index with dynamic fields unstored. +
      • + +
    • + Simple Text Query on Non-Default Index, specifying an index + that consists only of content derived from a specific field from a + specific document-type. +
      • + +
    • + Simple Text Query on Stored Field, specifying the field to be searched; targeting an index with dynamic + fields stored, to ensure that field-content is included in the return object. +
      • + +
    • + Match Query with Facet, showing how query-results can be displayed + either by row or by hits; + and demonstrating use of a facet, which provides aggregation-data. +
      • + +
    • + DocId Query, showing results of a query on two document IDs. +
      • + +
    • + Unanalyzed Term Query with Fuzziness Level of 0, demonstrating how to query on a term with no analysis. + Zero fuzziness is specified, to ensure + that matches are exact. +
      • + +
    • + Unanalyzed Term Query with Fuzziness Level of 2, which is almost identical to the immediately preceding query; but which this time + specifies a fuzziness factor of 2, allowing partial matches + to be made. The + output from this query can be compared to that of the one immediately preceding. +
      • + +
    • + Match Phrase Query, using Analysis, for searching on a phrase. +
      • + +
    • + Phrase Query, without Analysis, for searching on a phrase without analysis supported. +
      • + +
    • + Query String Query, showing how a query string is specified as search-input. +
      • + +
    • + Conjunction Query, whereby two separate queries are defined and then run as part of the search, + with only the matches returned by both included in the result-object. +
      • + + +
    • + Wild Card Query, whereby a wildcard is used in the string submitted for the search. +
      • + +
    • + Numeric Range Query, whereby minimum and maximum numbers are specified, and matches + within the range returned. +
      • + +
    • + Regexp Query, whereby a regular expression is submitted, to generate the conditions for + successful matches. +
      • + +
    + package main + +import ( + "encoding/json" + "fmt" + + "github.com/couchbase/gocb" + "github.com/couchbase/gocb/cbft" +) + +func simpleTextQuery(b *gocb.Bucket) { + indexName := "travel-sample-index-unstored" + query := gocb.NewSearchQuery(indexName, cbft.NewMatchQuery("swanky")). + Limit(10) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Simple Text Query Error:", err.Error()) + } + + printResult("Simple Text Query", result) +} + +func simpleTextQueryOnStoredField(b *gocb.Bucket) { + indexName := "travel-sample-index-stored" + query := gocb.NewSearchQuery(indexName, + cbft.NewMatchQuery("MDG").Field("destinationairport")). + Limit(10).Highlight(gocb.DefaultHighlightStyle) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Simple Text Query on Stored Field Error:", err.Error()) + } + + printResult("Simple Text Query on Stored Field", result) +} + +func simpleTextQueryOnNonDefaultIndex(b *gocb.Bucket) { + indexName := "travel-sample-index-hotel-description" + query := gocb.NewSearchQuery(indexName, cbft.NewMatchQuery("swanky")). + Limit(10) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Simple Text Query on Non-Default Index Error:", err.Error()) + } + + printResult("Simple Text Query on Non-Default Index", result) +} + +func textQueryOnStoredFieldWithFacet(b *gocb.Bucket) { + indexName := "travel-sample-index-stored" + query := gocb.NewSearchQuery(indexName, cbft.NewMatchQuery("La Rue Saint Denis!!"). + Field("reviews.content")).Limit(10).Highlight(gocb.DefaultHighlightStyle). + AddFacet("Countries Referenced", cbft.NewTermFacet("country", 5)) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Match Query with Facet, Result by Row Error:", err.Error()) + } + + printResult("Match Query with Facet, Result by hits:", result) + + fmt.Println() + fmt.Println("Match Query with Facet, Result by facet:") + for _, row := range result.Facets() { + jRow, err := json.Marshal(row) + if err != nil { + fmt.Println("Print Error:", err.Error()) + } + fmt.Println(string(jRow)) + } +} + +func docIdQueryMethod(b *gocb.Bucket) { + indexName := "travel-sample-index-unstored" + query := gocb.NewSearchQuery(indexName, cbft.NewDocIdQuery("hotel_26223", "hotel_28960")) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("DocId Query Error:", err.Error()) + } + + printResult("DocId Query", result) +} + +func unAnalyzedTermQuery(b *gocb.Bucket, fuzzinessLevel int) { + indexName := "travel-sample-index-stored" + query := gocb.NewSearchQuery(indexName, cbft.NewTermQuery("sushi").Field("reviews.content"). + Fuzziness(fuzzinessLevel)).Limit(50).Highlight(gocb.DefaultHighlightStyle) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Printf("Unanalyzed Term Query with Fuzziness Level of %d Error: %s\n", fuzzinessLevel, err.Error()) + } + + printResult(fmt.Sprintf("Unanalyzed Term Query with Fuzziness Level of %d", fuzzinessLevel), result) +} + +func matchPhraseQueryOnStoredField(b *gocb.Bucket) { + indexName := "travel-sample-index-stored" + query := gocb.NewSearchQuery(indexName, + cbft.NewMatchPhraseQuery("Eiffel Tower").Field("description")). + Limit(10).Highlight(gocb.DefaultHighlightStyle) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Match Phrase Query, using Analysis Error:", err.Error()) + } + + printResult("Match Phrase Query, using Analysis", result) +} + +func unAnalyzedPhraseQuery(b *gocb.Bucket) { + indexName := "travel-sample-index-stored" + query := gocb.NewSearchQuery(indexName, + cbft.NewPhraseQuery("dorm", "rooms").Field("description")). + Limit(10).Highlight(gocb.DefaultHighlightStyle) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Phrase Query, without Analysis Error:", err.Error()) + } + + printResult("Phrase Query, without Analysis", result) +} + +func conjunctionQueryMethod(b *gocb.Bucket) { + indexName := "travel-sample-index-stored" + firstQuery := cbft.NewMatchQuery("La Rue Saint Denis!!").Field("reviews.content") + secondQuery := cbft.NewMatchQuery("boutique").Field("description") + + conjunctionQuery := cbft.NewConjunctionQuery(firstQuery, secondQuery) + + result, err := b.ExecuteSearchQuery(gocb.NewSearchQuery(indexName, conjunctionQuery). + Limit(10).Highlight(gocb.DefaultHighlightStyle)) + if err != nil { + fmt.Println() + fmt.Println("Conjunction Query Error:", err.Error()) + } + + printResult("Conjunction Query", result) +} + +func queryStringMethod(b *gocb.Bucket) { + indexName := "travel-sample-index-unstored" + query := gocb.NewSearchQuery(indexName, cbft.NewQueryStringQuery("description: Imperial")). + Limit(10) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Query String Query Error:", err.Error()) + } + + printResult("Query String Query", result) +} + +func wildCardQueryMethod(b *gocb.Bucket) { + indexName := "travel-sample-index-stored" + query := gocb.NewSearchQuery(indexName, cbft.NewWildcardQuery("bouti*ue").Field("description")). + Limit(10).Highlight(gocb.DefaultHighlightStyle) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Wild Card Query Error:", err.Error()) + } + + printResult("Wild Card Query", result) +} + +func numericRangeQueryMethod(b *gocb.Bucket) { + indexName := "travel-sample-index-unstored" + query := gocb.NewSearchQuery(indexName, cbft.NewNumericRangeQuery(). + Min(10100, true).Max(10200, true).Field("id")). + Limit(10).Highlight(gocb.DefaultHighlightStyle) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Wild Card Query Error:", err.Error()) + } + + printResult("Wild Card Query", result) +} + +func regexpQueryMethod(b *gocb.Bucket) { + indexName := "travel-sample-index-stored" + query := gocb.NewSearchQuery(indexName, cbft.NewRegexpQuery("[a-z]"). + Field("description")).Limit(10).Highlight(gocb.DefaultHighlightStyle) + + result, err := b.ExecuteSearchQuery(query) + if err != nil { + fmt.Println() + fmt.Println("Regexp Query Error:", err.Error()) + } + + printResult("Regexp Query", result) +} + +func printResult(label string, results gocb.SearchResults) { + fmt.Println() + fmt.Println("= = = = = = = = = = = = = = = = = = = = = = =") + fmt.Println("= = = = = = = = = = = = = = = = = = = = = = =") + fmt.Println() + fmt.Println(label) + fmt.Println() + + for _, row := range results.Hits() { + jRow, err := json.Marshal(row) + if err != nil { + fmt.Println("Print Error:", err.Error()) + } + fmt.Println(string(jRow)) + } +} + +func main() { + cluster, err := gocb.Connect("couchbase://10.111.180.101") + if err != nil { + panic("error conencting to cluster:" + err.Error()) + } + cluster.Authenticate(gocb.PasswordAuthenticator{ + Username: "Administrator", + Password: "password", + }) + bucket, err := cluster.OpenBucket("travel-sample", "") + if err != nil { + panic("error opening bucket:" + err.Error()) + } + + simpleTextQuery(bucket) + simpleTextQueryOnStoredField(bucket) + simpleTextQueryOnNonDefaultIndex(bucket) + textQueryOnStoredFieldWithFacet(bucket) + docIdQueryMethod(bucket) + unAnalyzedTermQuery(bucket, 0) + unAnalyzedTermQuery(bucket, 2) + matchPhraseQueryOnStoredField(bucket) + unAnalyzedPhraseQuery(bucket) + conjunctionQueryMethod(bucket) + queryStringMethod(bucket) + wildCardQueryMethod(bucket) + numericRangeQueryMethod(bucket) + regexpQueryMethod(bucket) + + cluster.Close() +} + + +
    + diff --git a/content/sdk/go/handling-error-conditions.dita b/content/sdk/go/handling-error-conditions.dita index 3bffbcbc4e..bb029445f0 100644 --- a/content/sdk/go/handling-error-conditions.dita +++ b/content/sdk/go/handling-error-conditions.dita @@ -20,6 +20,17 @@ te, ok := err.(interface { Temporary() bool }) return ok :& :& te.Temporary() }

    + +
    + Enhanced Errors +

    The Go SDK has been updated to support Enhanced Error messages. Enhanced Errors are enabled + at the cluster level, before opening a bucket, with cluster.SetEnhancedErrors(true).

    +

    If you enable this, the errors that you get back from the library may not be directly comparable + to the gocb.Err*** constants, as they may have been dynamically generated to + include additional error information. You can access the underlying error using the new API + gocb.ErrorCause(err) to receive the underlying constants-comparable error object.

    +
    +
    diff --git a/content/sdk/go/managing-clusters.dita b/content/sdk/go/managing-clusters.dita index b2489b5d47..aabf4345cc 100644 --- a/content/sdk/go/managing-clusters.dita +++ b/content/sdk/go/managing-clusters.dita @@ -61,7 +61,7 @@ err := manager.UpdateBucket(bucketSettings) Flushing Buckets

    You may flush a bucket in the Go SDK by using the - BucketManager.Flush() method.

    + BucketManager.Flush() method (for an example of connecting to the BucketManager, see the Index Management section below).

    The flush operation may fail if the bucket does not have flush enabled:{ @@ -77,7 +77,7 @@ err := manager.UpdateBucket(bucketSettings) indexing fields "foo" and "bar":

    myCluster, _ := gocb.Connect("couchbase://localhost") testBucket, _ := myCluster.OpenBucket("test") -testManager := testBucket.BucketManager() +testManager := testBucket.Manager(username, password) //true, false at the end: ignore errors if the index exists, don't defer the creation testManager.CreateIndex("fooBar", string[]{ "foo", "bar" }, true, false); diff --git a/content/sdk/go/managing-connections.dita b/content/sdk/go/managing-connections.dita index 3901878799..42ec64e426 100644 --- a/content/sdk/go/managing-connections.dita +++ b/content/sdk/go/managing-connections.dita @@ -19,6 +19,13 @@ comma:cluster, err := gocb.Connect("couchbase://host1,host2,host3,hostN")Specifying multiple hosts may add additional redundancy when bootstrapping.

    + +
    + Authenticating +

    From Couchbase Server 5.0, you will need to authenticate the user, rather than against the bucket, as part of Role-Based Access Control. You will need to use PasswordAuthenticator:cluster.Authenticate(gocb.PasswordAuthenticator{ + Username: username, + Password: password, + })

    Opening a Bucket

    Once the Cluster object has been created, you may open one or more @@ -27,7 +34,7 @@ - which should be provided if the bucket is password protected, or supplied as nil otherwise.

    bucket1, err := cluster.OpenBucket("messages", nil) -protectedBucket, err := cluster.openBucket("protected", "s3cr3t") +protectedBucket, err := cluster.OpenBucket("protected", "s3cr3t")

    Note that the bucket password is not the administrative password used to access the Couchbase Web UI.

    Once the bucket has been created, it may be used throughout your application. Buckets diff --git a/content/sdk/go/sample-app-backend.dita b/content/sdk/go/sample-app-backend.dita index 370dc6b69e..439d7195b2 100644 --- a/content/sdk/go/sample-app-backend.dita +++ b/content/sdk/go/sample-app-backend.dita @@ -8,7 +8,7 @@

    The full source code for the tutorial is available on GitHub couchbaselabs/try-cb-golang. The primary focus of the tutorial is to explain the function and theory behind the Couchbase Java client and how it works together with Couchbase Server, and especially new features in versions 4.0/4.5 like N1QL, FTS and sub-document. It makes use of the travel-sample data set. The code that generates the web application is provided with the source code, @@ -38,13 +38,8 @@ This tutorial focuses on querying through N1QL and FTS rather than - views. If you want information about using views, see the following resources:

      -
    • General Information about views: -
      • -
    • Querying views:
      • -
    + views. If you want information about using views, see + Views.
    Walking Through the API @@ -69,6 +64,12 @@ globalCluster, err = gocb.Connect(cbConnStr) // error handling elided... +cluster.Authenticate(gocb.PasswordAuthenticator{ + Username: "Administrator", + Password: "something", +}) +// For RBAC, in Couchbase Server 5.0 and up + globalBucket, err = globalCluster.OpenBucket(cbBucket, cbPassword) // error handling elided...

    Both bucket and cluster can be managed through the SDK as well (eg. add views or @@ -304,9 +305,9 @@ rows, err := globalBucket.ExecuteSearchQuery(q)The res.Content("description", &hotel.Description) respData.Data = append(respData.Data, hotel) }Each - FTS result is represented as an ISearchQueryRow which exposes the + FTS result is represented as an SearchRowHit which exposes the document's Id. The sub-document API can then be used to fetch data - (bucket.LookupIn<T>(documentId)) and specify what parts it + (bucket.LookupIn(documentId)) and specify what parts it wants: name, description, address, city, state and country. The application then Execute() the sub-document query. In the rest of the code, the address-related fields are aggregated together and the data obtained is returned as a diff --git a/content/sdk/go/sdk-authentication-overview.dita b/content/sdk/go/sdk-authentication-overview.dita index 2c3df9ca21..b0c0db87c2 100644 --- a/content/sdk/go/sdk-authentication-overview.dita +++ b/content/sdk/go/sdk-authentication-overview.dita @@ -85,8 +85,7 @@

  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    @@ -100,7 +99,51 @@

    • + +
    + + Certificate-Based Authentication + + +

    + Couchbase Server supports the use of X.509 certificates to authenticate clients (only available in + the Enterprise Edition, not the Community Edition). This allows + authenticated users to access specific resources by means of the data service (no other service + is currently available to clients through this form of authentication). +

    + + +

    + The process relies on a certificate + authority, for the issuing of certificates that validate identities. A certificate includes + information such as the name of the entity it identifies, an expiration date, the name of the authority + that issued the certificate, and the digital signature of the authority. + A client attempting to access Couchbase Server can present a certificate to the server, allowing + the server to check the validity of the certificate. If the certificate is valid, the user + under whose identity the client is running, + and the roles assigned that user, are verified. If the assigned roles are appropriate for + the level of access requested to the specified resource, access is granted. +

    + +

    The Go SDK has supported certificate authentication since version 1.3.3. The client certificate can be generated with the shell script on + the C SDK Authentication page. + Connection involves using gocb.CertAuthenticator{} to tell the Server that cert-auth is the + authenticator, and passing certpath and keypath in the connection string: +cluster, err := gocb.Connect("couchbases://10.0.0.1?certpath=/path/to/chain.pem&keypath=/path/to/client.key") +if err != nil { + panic(err) +} +cluster.Authenticate(gocb.CertAuthenticator{}) +bucket, err := cluster.OpenBucket("travel-sample", "") +if err != nil { + panic(err) +}

    + If the certificate generated does not directly match the host, then the SAN (Subject Alternative Name) field should supply the + appropriate names: gocb may fail to authenticate if it can't verify the hostname is valid for the certificate. Other SDKs' platforms + - Java and .NET - are by default less strict on verification of hostnames. +
    + diff --git a/content/sdk/go/sdk-user-management-overview.dita b/content/sdk/go/sdk-user-management-overview.dita index 7d77dc8ba9..9ba8d0aecc 100644 --- a/content/sdk/go/sdk-user-management-overview.dita +++ b/content/sdk/go/sdk-user-management-overview.dita @@ -119,8 +119,8 @@
  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles + .

    diff --git a/content/sdk/go/sdk-xattr-example.dita b/content/sdk/go/sdk-xattr-example.dita index c3848ba0f4..35a9d1ba83 100644 --- a/content/sdk/go/sdk-xattr-example.dita +++ b/content/sdk/go/sdk-xattr-example.dita @@ -50,7 +50,7 @@ import ( func main() { cluster, _ := gocb.Connect("couchbase://localhost") cluster.Authenticate(gocb.PasswordAuthenticator{ - Username: "Administrator",main.go + Username: "Administrator", Password: "password", }) diff --git a/content/sdk/go/start-using-sdk.dita b/content/sdk/go/start-using-sdk.dita index 75bc5abd1a..71720da3df 100644 --- a/content/sdk/go/start-using-sdk.dita +++ b/content/sdk/go/start-using-sdk.dita @@ -40,8 +40,8 @@ type User struct { func main() { cluster, _ := gocb.Connect("couchbase://localhost") cluster.Authenticate(gocb.PasswordAuthenticator{ - Username: ‘USERNAME’, - Password: ‘PASSWORD’, + Username: "USERNAME", + Password: "PASSWORD", }) bucket, _ := cluster.OpenBucket("bucketname", "") diff --git a/content/sdk/go/threshold-logging.dita b/content/sdk/go/threshold-logging.dita new file mode 100644 index 0000000000..813f31616b --- /dev/null +++ b/content/sdk/go/threshold-logging.dita @@ -0,0 +1,47 @@ + + + + Threshold Logging Tracing through the SDK + Threshold Logging + + + +

    +
    + +
    +
    + +
    +
    + + +
    + + Threshold Logging in Go + +

    + Response Time Observability is implemented in the Go SDK from version 1.4 onwards. + With OperationTracingEnabled set to true [the default in Couchbase Server 5.5], operation_tracing is passed in the cluster + connection query string. And with OperationTracingServerDurationEnabled set [it + currently defaults to false], the server_duration is passed in the cluster connection + query string. This is a brief example of customizing the threshold settings.cluster, _ := gocb.Connect("couchbase://localhost”) +// cluster, _ := gocb.Connect("couchbase://localhost?operation_tracing=true&server_duration=true”) example of using connection string options +cluster.Authenticate(gocb.PasswordAuthenticator{ + Username: username, + Password: password, +}) + +tracer := gocb.ThresholdLoggingTracer{} +tracer.Interval = 1 * time.Second +tracer.KvThreshold = 500 * time.Millisecond +tracer.ViewsThreshold = 1 * time.Second +tracer.N1qlThreshold = 1 * time.Second +tracer.SearchThreshold = 1 * time.Second +tracer.AnalyticsThreshold = 1 * time.Second +tracer.SampleSize = 20 +cluster.SetTracer(&tracer) // Note that this must come before cluster.OpenBucket +

    +
    + + \ No newline at end of file diff --git a/content/sdk/go/tracing-from-the-sdk.dita b/content/sdk/go/tracing-from-the-sdk.dita new file mode 100644 index 0000000000..f1d9adf3bc --- /dev/null +++ b/content/sdk/go/tracing-from-the-sdk.dita @@ -0,0 +1,119 @@ + + + + Tracing from the Go SDK with Couchbase Server + Tracing from the SDK + Use tracing from the SDK to discover timeouts and bottlenecks across the netowrk and cluster. + +

    For our example, we will customize the threshold logging tracer settings with CouchbaseEnvironment.Builder. By default it + will log every minute (if something is found) and only sample the top 10 slowest operations. The default threshold for key/value operation is 500 milliseconds. We shall + set them so low that almost everything gets logged - not something you should do in production!tracer := gocb.ThresholdLoggingTracer{} +tracer.Interval = 1 * time.Second // log every second +tracer.KvThreshold = 1 * time.Microsecond +Tracer.SampleSize = 100000 // log everythingThis will set our threshold for key/value operations to one microsecond, log the found operations every second, and set the sample size to a very large value so everything will be logged.

    + +

    With these configs in place we are ready to run some operations. Below, we read some documents from the travel-sample bucket and, if found, write them back with an upsert - giving us both read and write operations to log.// Set the logger so that we see the output +gocb.SetLogger(gocb.DefaultStdioLogger()) + +// Connect +cluster, _ := gocb.Connect("couchbase://localhost”) +cluster.Authenticate(gocb.PasswordAuthenticator{ + Username: username, + Password: password, +}) + +cluster.SetTracer(&tracer) +bucket, _ := cluster.OpenBucket(“travel-sample”, “”) + +// Loads some docs and write them back +for i := 0; i < 5; i++ { + airline := struct { + Type string `json:"type"` + ID uint `json:"id"` + }{} + + key := fmt.Sprintf("airline_1%d", i) + cas, err := bucket.Get(key, &airline) + if err == nil { + bucket.Replace(key, airline, cas, 0) + } +} + +// Leave a bit of time for the logger to output +time.Sleep(5 * time.Second)

    + +

    Run the code, and you will see something like the following in the logs:{ + "service": "kv", + "count": 6, + "top": [ + { + "operation_name": "Get", + "total_us": 31903, + "server_us": 22, + "decode_us": 91, + "last_operation_id": "0x5”, + "last_local_id": "2f87cf1799fe15fd\/cda32b624efa3e53" + }, + { + "operation_name": "Get", + "total_us": 1204, + "server_us": 8, + "last_operation_id": "0x7", + "last_local_id": "2f87cf1799fe15fd\/cda32b624efa3e53" + }, + { + "operation_name": "Get", + "total_us": 1166, + "server_us": 14, + "last_operation_id": "0xa", + "last_local_id": "2f87cf1799fe15fd\/cda32b624efa3e53" + }, + { + "operation_name": "Replace", + "total_us": 422, + "encode_us": 15, + "server_us": 49, + "last_operation_id": "0x6", + "last_local_id": "2f87cf1799fe15fd\/cda32b624efa3e53" + }, + { + "operation_name": "Get", + "total_us": 407, + "server_us": 8, + "last_operation_id": "0x9", + "last_local_id": "2f87cf1799fe15fd\/cda32b624efa3e53" + }, + { + "operation_name": "Get", + "total_us": 385, + "server_us": 8, + "last_operation_id": "0x8", + "last_local_id": "2f87cf1799fe15fd\/cda32b624efa3e53" + } + ] +}For each service (only kv-based on this workload), the threshold log reporter will show you the total number of recorded ops (through count), and give you the top slowest ops sorted by their latency. Since only airline_10 exists in the bucket you will see five document fetches but only one mutation.

    + + +
    + + Output fields in detail. + +

    Let's highlight a single operation, and explain each field in a little more detail:{ + "operation_name": "Get", + "total_us": 31903, + "server_us": 22, + "decode_us": 91, + "last_operation_id": "0x5”, + "last_local_id": "2f87cf1799fe15fd\/cda32b624efa3e53" +}

    +

    This tells us the following:

      +
    • total_us: The total time it took to perform the full operation: here around 32 milliseconds.
      • +
    • server_us: The server reported that its work performed took 22 microseconds (this does not include network time or time in the buffer before picked up at the cluster).
      • +
    • decode_us: Decoding the response took the client 91 microseconds.
      • +
    • last_operation_id: A combination of type of operation and id (in this case the opaque value), useful for diagnosing and troubleshooting in combination with the last_local_id.
      • +
    • last_local_id: With Server 5.5 and later, this id is negotiated with the server and can be used to correlate logging information on both sides in a simpler fashion.

    +

    You can see that if the thresholds are set the right way based on production requirements, without much effort slow operations can be logged and pinpointed more easily than before.

    +
    + + +     diff --git a/content/sdk/java.ditamap b/content/sdk/java.ditamap index ca7856789d..1eb26cb2c2 100644 --- a/content/sdk/java.ditamap +++ b/content/sdk/java.ditamap @@ -19,6 +19,9 @@ + + + @@ -27,7 +30,9 @@ - + + + @@ -39,6 +44,7 @@ + @@ -50,5 +56,7 @@ + + diff --git a/content/sdk/java/async-programming.dita b/content/sdk/java/async-programming.dita index 611c266820..5582456aca 100644 --- a/content/sdk/java/async-programming.dita +++ b/content/sdk/java/async-programming.dita @@ -575,9 +575,10 @@ Sum: 15]]>

    Combined with the Java SDK, this technique can be used to separate returned Documents based on their content. The following example uses a view to load all documents from the beer-sample bucket, groups them by type and counts the number of occurrences:

    document.content().getString("type")) .subscribe(observable -> observable.count().subscribe(integer -> diff --git a/content/sdk/java/client-settings.dita b/content/sdk/java/client-settings.dita index 23e6282327..51b015b417 100644 --- a/content/sdk/java/client-settings.dita +++ b/content/sdk/java/client-settings.dita @@ -546,7 +546,7 @@ CouchbaseCluster cluster = CouchbaseCluster.create(environment);]]> Endpoints(int) query \ Endpoints - 1 + The default is dynamic mode, please see next row. The number of actual endpoints (sockets) to open per Node in the cluster against the Query (N1QL) service. By default only one socket is opened to avoid unnecessary wasting resources. If you plan to run a query heavy workload, especially @@ -565,8 +565,8 @@ CouchbaseCluster cluster = CouchbaseCluster.create(environment);]]> ServiceConfig(int, int) query \ ServiceConfig - Need to set minimum and maximum number of Endpoints; and optionally the minimum - idle time, in seconds, after which the socket will be closed. + Need to set minimum and maximum number of Endpoints; and optionally the minimum idle time, in seconds, after which the socket will be closed. + The default minimum and maximum are 1 and 12 respectively." To allow dynamic pooling, as an alternative to the fixed values of queryEndPoints. Note that if both queryEndpoints and queryServiceConfig are set, queryEndpoints takes priority. This helps to ensure backwards compatibility. @@ -592,7 +592,7 @@ CouchbaseCluster cluster = CouchbaseCluster.create(environment);]]> or search \ ServiceConfig(int, int) search \ServiceConfig - >Need to set minimum and maximum number of Endpoints; and optionally the minimum + Need to set minimum and maximum number of Endpoints; and optionally the minimum idle time, in seconds, after which the socket will be closed. To allow dynamic pooling, as an alternative to the fixed values of searchEndPoints. Note that if both searchEndpoints and searchServiceConfig are set, queryEndpoints takes diff --git a/content/sdk/java/compatibility-versions-features.dita b/content/sdk/java/compatibility-versions-features.dita index 16f7d6bc94..8a1d035ca8 100644 --- a/content/sdk/java/compatibility-versions-features.dita +++ b/content/sdk/java/compatibility-versions-features.dita @@ -1,6 +1,6 @@ - + Compatibility of Couchbase Features, Couchbase Versions and the Couchbase Java SDK Compatibility @@ -8,12 +8,12 @@ Couchbase Version/SDK Version Matrix

    Couchbase SDKs are tested against a variety of different environments to ensure both backward and forward compatibility with different versions of Couchbase Server. The matrix - below denotes the version of Couchbase Server, the version of the Java SDK and whether the SDK - is:

      + below denotes the version of Couchbase Server, the version of the Java SDK, and whether the SDK + is:
      • ✖ Unsupported: This combination is not tested, and is not within the scope of technical support if you have purchased a support agreement.
      • ◎ Compatible: This combination has been tested previously, and should be - compatible. This combination is not supported or recommended by our technical support + compatible. This combination is not recommended by our technical support organization. It is best to upgrade either the SDK or the Couchbase version you are using.
      • ✔ Supported:This combination is subject to ongoing quality assurance, and is @@ -21,90 +21,46 @@

      Recommended SDK per Server Version Matrix - + - - Under 1.4 - SDK 1.4 - SDK 2.0, 2.1 - SDK 2.2, 2.3 - SDK 2.4 + SDK 2.0 + SDK 2.1 + SDK 2.2, 2.3, 2.4 SDK 2.5 - SDK released for > - N/A (Archived) - Bug fixes only - N/A (Archived) - Bug fixes and minor features Back ports - New Features, Active Development - - - Server 1.8 - ◎◎◎ + Server 3.1 + ◎ ◎ ◎ - ◎ + ✖ - Server 2.0 - ◎◎◎ - ◎ ◎ - ◎ - - - Server 2.5 - ◎◎◎ - ◎ ◎ - ◎ - - - Server 3.0 - ◎◎◎ - ✔ ✔ - ✔ - - - Server 4.0 - ◎◎◎ - ✔ ✔ - ✔ - - - Server 4.1 - ◎◎◎ - ✔ ✔ - ✔ - - - Server 4.5 - ◎◎◎ - ✔ ✔ - ✔ - - - Server 4.6 - ◎◎◎ - ✔ ✔ + Server 4.0-4.6 + ◎ + ◎ ✔ ✔ - Server 5.0 - ◎◎◎ + Server 5.0-5.1 + ◎ ◎ ✔ ✔
      +

      Note the + End of Life dates for Couchbase Server and SDK versions. + See the notes there for Support details.

      To take advantage of all features offered by Couchbase Server, you need to know what version of the Java SDK provides compatibility for the features you want to use. The @@ -112,7 +68,7 @@ features of each version of Couchbase Server.

      Couchbase Server and SDK Supported Version Matrix - + @@ -120,6 +76,7 @@ + @@ -129,80 +86,104 @@ Server 3.0, 3.1 Server 4.0, 4.1 Server 4.5, 4.6 + Server 5.0 - Basic Features + Basic Features CRUD Operations - Since 1.0 + Since 1.0 View Querying APIs - Since 1.1 + Since 1.1 Geospatial Views Not Supported 1.1 - 1.4 - Since 2.1 + Since 2.1 - Advanced Features + Advanced Features Durability Requirements - Since 1.1 + Since 1.1 Carrier Publication Configuration Not Supported - Since 1.4 + Since 1.4 SSL Connectivity Not Supported - Since 2.0 + Since 2.0 Bulk Operations - Since 1.0 + Since 1.0 N1QL Querying Since 2.0 (experimental) - Since 2.2 + Since 2.2 Multi-Dimensional Scaling Not Supported - Since 2.2 + Since 2.2 Sub-document API Not Supported - Since 2.3 + Since 2.3 - Fulltext Search + Full Text Search + Not Supported + Since 2.3 (Experimental) + Since 2.4 + + + Administrative Features + + + + Datastructures Not Supported - Since 2.3 (experimental) + Since 2.3 + - Administrative Features + Extended Attributes + Not Supported + Since 2.4 Administrative API - Since 1.0 + Since 1.0 + + + + RBAC + Not Supported + Since 2.4 +
      + With RBAC, + in Couchbase Server 5.0, authentication will depend upon access privileges, and upgrading to 5.0 + with existing buckets, or upgrading Server, will require taking these changes into account.
    • JDK Version Compatibility @@ -232,8 +213,8 @@ - SDC 1.4.x - SDC 2.1 & 2.2 + SDC 2.1.x + SDC 2.2 & 3.0 @@ -243,48 +224,18 @@ New Features, Active Development - Server 1.8 + Server 3.1 Compatible Recommended - Server 2.0 + Server 4.0-4.6 Compatible Recommended - - Server 2.5 - Compatible - Recommended - - - Server 3.0 - Compatible - Recommended - - - Server 4.0 - Compatible (no N1QL) - Recommended - - - Server 4.1 - Compatible (no N1QL) - Recommended - - - Server 4.5 - Compatible (no N1QL) - Recommended - - - Server 4.6 - Compatible (no N1QL) - Recommended - Server 5.0 - Compatible (no N1QL) + Compatible Recommended diff --git a/content/sdk/java/encrypting-using-sdk.dita b/content/sdk/java/encrypting-using-sdk.dita new file mode 100644 index 0000000000..0667088788 --- /dev/null +++ b/content/sdk/java/encrypting-using-sdk.dita @@ -0,0 +1,61 @@ + + + Field Level Encryption from the Java SDK + Field Level Encryption is available in Couchbase Data Platform 5.5, from Java SDK version 2.6.0 + +
    + Packaging +

    The Couchbase Java SDK uses + the java-couchbase-encryption library + to provide support for encryption and decryption of JSON fields.

    + +

    The Couchbase Java Field Level Encryption (FLE) uses entity annotations/JsonObject methods to specify which field(s) to apply encryption and which algorithm to use. Here’s an example POJO entity definition:public static class Person { + + @Id + public String id; + + @EncryptedField(provider = "AES") + public String password; + + //The rest will be transported and stored unencrypted + public String firstName; + public String lastName; + public String userName; + public int age; + }

    +
    + + +
    + Configuration +

    Create a configuration to connect to your Couchbase cluster and configure the crypto manager in the Couchbase environment with the encryption crypto algorithm and key store providers.The alias used for registering the provider should match the provider name string in the annotation/parameter.JceksKeyStoreProvider kp = new JceksKeyStoreProvider("secret"); +kp.storeKey("SecretKey", "mysecretkey".getBytes()); +kp.storeKey("HMACsecret", "myauthsecret".getBytes()); +kp.publicKeyName("secret_key"); +kp.signingKeyName("hmac_key"); +AES256CryptoProvider aes256CryptoProvider = new AES256CryptoProvider(kp); +CryptoManager cryptoManager = new CryptoManager(); +cryptoManager.registerProvider("AES", aes256CryptoProvider); +CouchbaseEnvironment environment = DefaultCouchbaseEnvironment.builder().cryptoManager(cryptoManager).build();

    + +

    To apply encryption to a repository entity, use the EncryptedField annotation with provider name as registered on the crypto manager.@EncryptedField(provider = "AES") +public String password; +

    +

    Encryption can also be applied to JsonObjects.JsonObject.create().putAndEncrypt("foo", "bar", "AES"); +

    +
    + + +
    + Decryption +

    Encrypted fields in entities are decrypted based on the annotations. To decrypt use fields + in JsonObject, use getAndDecrypt methods.JsonDocument stored = bucket.get("mydoc"); +stored.content().getAndDecrypt("foo", "AES") +

    +
    + + + +     +     \ No newline at end of file diff --git a/content/sdk/java/full-text-searching-with-sdk.dita b/content/sdk/java/full-text-searching-with-sdk.dita index 6a5e878eca..de9c476030 100644 --- a/content/sdk/java/full-text-searching-with-sdk.dita +++ b/content/sdk/java/full-text-searching-with-sdk.dita @@ -1,108 +1,463 @@ - Full Text Search (FTS) Using the Java SDK with Couchbase Server - Searching from the SDK - You can use the Full Text Search service (FTS) to create queryable full-text indexes in Couchbase Server. + + Searching from the SDK + + + + + + Searching from the SDK + + + + + The Java SDK provides an API that supports Full Text Search. + + + -

    Couchbase offers Full-text search support, allowing you to search for documents that - contain certain words or phrases. In the Java SDK you can search full-text indexes by - using the Bucket.query(SearchQuery) API.

    - -

    Querying a FTS index through the Java client is performed through the Bucket.query(SearchQuery q) - method, providing a SearchQuery. Building a SearchQuery takes two - parameters, the index name to query and the actual search query itself (kind of a statement). Additional - search options may be specified by using the SearchQuer as a builder, chaining setters - for each relevant option.

    -

    This method returns a SearchQueryResult whose iterator yields the results of the query (in the - form of SearchQueryRow objects). It also exposes a status() for the request, - some execution metrics() and facets() results if some facets have been requested. - Instead of iterating directly on the result, you can access rows as a list through the hits() method, - and in case of execution errors you can inspect the error messages in the errors() method. Note that - partial results can happen in this case (and hits() will return them). Instead of getting partial results through - hits(), one can combine results and errors and get an exception through the use of hitsOrFail(). -

    -

    The SearchQueryRow object contains the index, id and - score properties, respectively identifying the exact FTS index that returned the hit, the id of the - document that matched and a decimal score for the match. It also contains optional sections depending on the request - and the availability of all relevant settings in the FTS mapping. Those are explanation() (an - explanation of the plan followed by the FTS index to execute the query), locations() (a map-like - listing of the location of all matching terms inside each relevant field that was queried), fragments() - (a map-like listing of occurrences of the search terms in each field, with the context of the terms) and - fields() (a map of the complete value of each requested field). Most of these need that the index be - configured to store the data of a searched field. -Bucket bkt = CouchbaseCluster.create("192.168.33.101").openBucket("travel-sample"); -MatchQuery fts = SearchQuery.match("term"); -SearchQueryResult result = bkt.query(new SearchQuery("travel-search", fts)); -for (SearchQueryRow row : result) { - System.out.println(row); -} -

    - -
    Query TypesThere are many different flavours of search queries, and each can be constructed - through static factory methods in the SearchQuery class. All of these types derive from the - AbstractFtsQuery and can be found in the - com.couchbase.client.java.search.queries.AbstractFtsQuery package. It contains query classes - corresponding to those enumerated in the FTS - generic documentation.

    It is important to distinguish between query options and general search - options. Some options affect the search process in general (such as the limit, indicating how - many results to return) while others only affect a specific query (such as fuzziness for a given - query). Because multiple queries can be combined in a single search operation, query specific options can be - specified only in the query object itself, while search options are specified at the level of the - SearchQuery class, using builder methods. -Bucket bkt = CouchbaseCluster.create("192.168.33.101").openBucket("travel-sample"); -MatchQuery fts = SearchQuery.match("term") - //query options: - .fuzziness(2).field("content"); -SearchQuery query = new SearchQuery("travel-search", fts) - //search options: - //will show value for activity and country fields - .fields("activity", "country") - //will have max 3 hits - .limit(3); - -SearchQueryResult result = bkt.query(query); -for (SearchQueryRow row : result) { - System.out.println(row); -}

    - -

    Here's some sample output for the previous query:

    - DefaultSearchQueryRow{index='travel-search_33760129d0737bff_b7ff6b68', id='landmark_11778', score=0.0313815325019958, explanation={}, \ -locations=DefaultHitLocations{size=3, locations=[HitLocation{field='content', term='tea', pos=39, start=254, end=257},HitLocation{field='content', \ -term='teas', pos=56, start=353, end=357},HitLocation{field='content', term='tart', pos=17, start=95, end=99}]}, fragments={}, fields={activity=eat, \ -country=United States}} - -DefaultSearchQueryRow{index='travel-search_33760129d0737bff_b7ff6b68', id='landmark_25547', score=0.02536160834515202, explanation={}, \ -locations=DefaultHitLocations{size=3, locations=[HitLocation{field='content', term='tea', pos=33, start=191, end=194},HitLocation{field='content', \ -term='try', pos=30, start=177, end=180},HitLocation{field='content', term='per', pos=57, start=337, end=340}]}, fragments={}, fields={activity=eat, \ -country=United States}} - -DefaultSearchQueryRow{index='travel-search_33760129d0737bff_8b80958a', id='landmark_26854', score=0.02079624734659704, explanation={}, \ -locations=DefaultHitLocations{size=10, locations=[HitLocation{field='content', term='trim', pos=227, start=1255, end=1259},HitLocation{field='content', \ -term='steam', pos=7, start=41, end=46},HitLocation{field='content', term='steam', pos=38, start=213, end=218},HitLocation{field='content', \ -term='steam', pos=74, start=424, end=429},HitLocation{field='content', term='steam', pos=93, start=532, end=537},HitLocation{field='content', \ -term='steam', pos=114, start=651, end=656},HitLocation{field='content', term='steam', pos=126, start=715, end=720},HitLocation{field='content', \ -term='steam', pos=145, start=819, end=824},HitLocation{field='content', term='steam', pos=300, start=1611, end=1616},HitLocation{field='content', \ -term='team', pos=59, start=335, end=339}]}, fragments={}, fields={activity=see, country=United States}} + +
    + + + Using Full Text Search from the Java SDK + + +

    + By means of the Java SDK, Full Text Search queries can be performed on + Full Text Indexes; and result-sets sorted. +

    + +

    + A general introduction to Full Text Search, with pointers to detailed descriptions of its + principal features, is provided in + Full Text Search: Fundamentals. +

    + +

    + The current page features a code example that demonstrates the Java SDK Full Text Search + API. The example assumes that Couchbase Server is running, and that + the username Administrator and + the password password provide authorization for performing the searches. + It also assumes that the travel-sample bucket + has been installed. + For information on creating users and managing roles, see + Authorization. + For information on installing sample buckets, see + Settings. +

    + +

    + The example also assumes the existence of three specific Full Text Indexes, defined on the + travel-sample bucket. These are: +

    + +
      +
    • + travel-sample-index-unstored: Uses only the default + settings. + +

      + +

      + +
      • + +
    • + travel-sample-index-stored: Uses default settings, with one exception: + dynamic fields are stored, for the whole index. + +

      + +

      + +
      • + +
    • + travel-sample-index-hotel-description: Indexes only the description + fields of hotel + documents, and disables the default type mapping. The index + has a custom analyzer named myUnicodeAnalyzer defined on it: the analyzer's + main characteristic is that it uses the unicode tokenizer. + +

      + +

      + +
      • +
    + +

    + See + Creating Indexes + for details on how to create these indexes: they can be created interactively, by + means of the Couchbase Web Console; however, there may be greater efficiency + in using the Couchbase REST + API, as described + in the section + Index Creation with the REST API. + The JSON objects that constitute index-definitions (for + inclusion as bodies to the index-creation REST calls), are provided in + Demonstration Indexes. +

    + +

    + The example features the following Full Text Searches on the + travel-sample bucket, within Couchbase Server: +

    + +
      + +
    • + Simple Text Query on a single word, targeting an index with dynamic fields unstored. + +

      + +

      +
      • + +
    • + Simple Text Query on Non-Default Index, specifying an index + that consists only of content derived from a specific field from a + specific document-type. + +

      + +

      +
      • + +
    • + Simple Text Query on Stored Field, specifying the field to be searched; targeting an index with dynamic + fields stored, to ensure that field-content is included in the return object. + +

      + +

      +
      • + +
    • + Match Query with Facet, showing how query-results can be displayed + either by row or by hits; + and demonstrating use of a facet, which provides aggregation-data. + +

      + +

      +
      • + +
    • + DocId Query, showing results of a query on two document IDs. + +

      + +

      +
      • + +
    • + Unanalyzed Term Query with Fuzziness Level of 0, demonstrating how to query on a term with no analysis. + Zero fuzziness is specified, to ensure + that matches are exact. +

      + +

      +
      • + +
    • + Unanalyzed Term Query with Fuzziness Level of 2, which is almost identical to the immediately preceding query; but which this time + specifies a fuzziness factor of 2, allowing partial matches + to be made. The + output from this query can be compared to that of the one immediately preceding. +

      + +

      +
      • + +
    • + Match Phrase Query, using Analysis, for searching on a phrase. +

      + +

      +
      • + +
    • + Phrase Query, without Analysis, for searching on a phrase without analysis supported. +

      + +

      +
      • + +
    • + Query String Query, showing how a query string is specified as search-input. +

      + +

      +
      • + +
    • + Conjunction Query, whereby two separate queries are defined and then run as part of the search, + with only the matches returned by both included in the result-object. +

      + +

      +
      • + + +
    • + Wild Card Query, whereby a wildcard is used in the string submitted for the search. +

      + +

      +
      • + +
    • + Numeric Range Query, whereby minimum and maximum numbers are specified, and matches + within the range returned. +

      + +

      +
      • + +
    • + Regexp Query, whereby a regular expression is submitted, to generate the conditions for + successful matches. +

      + +

      +
      • + +
    + +
    + + + Detailed Example + + +

    + The following example demonstrates Full Text Search queries that + can be made with the Java SDK. +

    + + package com.couchbase.example.fts; + +import com.couchbase.client.java.Bucket; +import com.couchbase.client.java.Cluster; +import com.couchbase.client.java.CouchbaseCluster; +import com.couchbase.client.java.search.SearchQuery; +import com.couchbase.client.java.search.facet.SearchFacet; +import com.couchbase.client.java.search.queries.*; +import com.couchbase.client.java.search.result.SearchQueryResult; +import com.couchbase.client.java.search.result.SearchQueryRow; + +public class JavaFtsExamples { + + public static void simpleTextQuery(Bucket bucket) { + String indexName = "travel-sample-index-unstored"; + MatchQuery query = SearchQuery.match("swanky"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10)); + + printResult("Simple Text Query", result); + } + + public static void simpleTextQueryOnStoredField(Bucket bucket) { + String indexName = "travel-sample-index-stored"; + MatchQuery query = SearchQuery.match("MDG").field("destinationairport"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10).highlight()); + + printResult("Simple Text Query on Stored Field", result); + } + + public static void simpleTextQueryOnNonDefaultIndex(Bucket bucket) { + String indexName = "travel-sample-index-hotel-description"; + MatchQuery query = SearchQuery.match("swanky"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10)); + + printResult("Simple Text Query on Non-Default Index", result); + } + + public static void textQueryOnStoredFieldWithFacet(Bucket bucket) { + String indexName = "travel-sample-index-stored"; + MatchQuery query = SearchQuery.match("La Rue Saint Denis!!") + .field("reviews.content"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10).highlight() + .addFacet("Countries Referenced", SearchFacet.term("country", 5))); + + printResult("Match Query with Facet, Result by Row", result); + + System.out.println(); + System.out.println("Match Query with Facet, Result by hits:"); + System.out.println(result.hits()); + + System.out.println(); + System.out.println("Match Query with Facet, Result by facet: "); + System.out.println(result.facets()); + } + + public static void docIdQueryMethod(Bucket bucket) { + String indexName = "travel-sample-index-unstored"; + DocIdQuery query = SearchQuery.docId("hotel_26223", "hotel_28960"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query)); + + printResult("DocId Query", result); + } + + public static void unAnalyzedTermQuery(Bucket bucket, int fuzzinessLevel) { + String indexName = "travel-sample-index-stored"; + TermQuery query = SearchQuery.term("sushi").field("reviews.content") + .fuzziness(fuzzinessLevel); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(50).highlight()); + + printResult("Unanalyzed Term Query with Fuzziness Level of " + + fuzzinessLevel + ":", result); + } + + public static void matchPhraseQueryOnStoredField(Bucket bucket) { + String indexName = "travel-sample-index-stored"; + MatchPhraseQuery query = SearchQuery.matchPhrase("Eiffel Tower") + .field("description"); -
    - Query Facets -

    Query facets may also be - added to the general search parameters by using the addFacet(String name, SearchFacet facet) - builder method on SearchQuery. You can create facet queries by instantiating facets through factory - methods in the SearchFacet class.

    - SearchQuery query = new SearchQuery("travel-search", fts) - //will have max 3 hits - .limit(3) - //will have a "category" facet on the top 3 countries in terms of hits - .addFacets(SearchFacet.term("countries", "country", 3)); -SearchQueryResult result = bkt.query(query); -System.out.println(result.facets()); - -

    Here is the facet part of the result from the query above:

    - {countries=TermFacetResult{name='countries', field='country', total=451, missing=0, other=0, terms=[{name='United States', \ - count=217}, {name='United Kingdom', count=188}, {name='France', count=46}]}} + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10).highlight()); + + printResult("Match Phrase Query, using Analysis", result); + } + + public static void unAnalyzedPhraseQuery(Bucket bucket) { + String indexName = "travel-sample-index-stored"; + PhraseQuery query = SearchQuery.phrase("dorm", "rooms") + .field("description"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10).highlight()); + + printResult("Phrase Query, without Analysis", result); + } + + public static void conjunctionQueryMethod(Bucket bucket) { + String indexName = "travel-sample-index-stored"; + MatchQuery firstQuery = SearchQuery.match("La Rue Saint Denis!!") + .field("reviews.content"); + MatchQuery secondQuery = SearchQuery.match("boutique") + .field("description"); + + ConjunctionQuery conjunctionQuery = + SearchQuery.conjuncts(firstQuery, secondQuery); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, conjunctionQuery).limit(10).highlight()); + + printResult("Conjunction Query", result); + } + + public static void queryStringMethod(Bucket bucket) { + String indexName = "travel-sample-index-unstored"; + QueryStringQuery query = SearchQuery.queryString("description: Imperial"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10)); + + printResult("Query String Query", result); + } + + public static void wildCardQueryMethod(Bucket bucket) { + String indexName = "travel-sample-index-stored"; + WildcardQuery query = SearchQuery.wildcard("bouti*ue").field("description"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10).highlight()); + + printResult("Wild Card Query", result); + } + + public static void numericRangeQueryMethod(Bucket bucket) { + String indexName = "travel-sample-index-unstored"; + NumericRangeQuery query = SearchQuery.numericRange().min(10100).max(10200) + .field("id"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10)); + + printResult("Numeric Range Query", result); + } + + public static void regexpQueryMethod(Bucket bucket) { + String indexName = "travel-sample-index-stored"; + RegexpQuery query = SearchQuery.regexp("[a-z]").field("description"); + + SearchQueryResult result = bucket.query( + new SearchQuery(indexName, query).limit(10).highlight()); + + printResult("Regexp Query", result); + } + + private static void printResult(String label, SearchQueryResult resultObject) { + System.out.println(); + System.out.println("= = = = = = = = = = = = = = = = = = = = = = ="); + System.out.println("= = = = = = = = = = = = = = = = = = = = = = ="); + System.out.println(); + System.out.println(label); + System.out.println(); + + for (SearchQueryRow row : resultObject) { + System.out.println(row); + } + } + + public static void main(String[] args) { + // Access the cluster that is running on the local host, authenticating with + // the username and password of any user who has the "FTS Searcher" role + // for the "travel-sample" bucket... + // + Cluster cluster = CouchbaseCluster.create("localhost"); + cluster.authenticate("Administrator", "password"); + Bucket travelSample = cluster.openBucket("travel-sample"); + + simpleTextQuery(travelSample); + + simpleTextQueryOnStoredField(travelSample); + + simpleTextQueryOnNonDefaultIndex(travelSample); + + textQueryOnStoredFieldWithFacet(travelSample); + + docIdQueryMethod(travelSample); + + unAnalyzedTermQuery(travelSample, 0); + + unAnalyzedTermQuery(travelSample, 2); + + matchPhraseQueryOnStoredField(travelSample); + + unAnalyzedPhraseQuery(travelSample); + + conjunctionQueryMethod(travelSample); + + queryStringMethod(travelSample); + + wildCardQueryMethod(travelSample); + + numericRangeQueryMethod(travelSample); + + regexpQueryMethod(travelSample); + + cluster.disconnect(); + } +}
    diff --git a/content/sdk/java/handling-error-conditions.dita b/content/sdk/java/handling-error-conditions.dita index 3c64db2e49..b15c8c05ba 100644 --- a/content/sdk/java/handling-error-conditions.dita +++ b/content/sdk/java/handling-error-conditions.dita @@ -873,6 +873,43 @@ while(true) { exponential or random delay.

    + +
    + +
    Retry for some exceptions
    +
    Using RetryBuilder allows a more nuanced approach, letting you mandate different responses to different exceptions, and leading to code which can be easier to reason about - like the two examples below. This first example configures the Retry Builder so that when an error happens during a document fetch, it retries up to 20 times and waits 100 ms in between attempts. But it only retries on the two exceptions given, everything else is propagated downstream. Note how the 100 * 20 ms = 2s aligns with the timeout provided; this doesn't have to be, but it makes sense to align it.bucket.async() +.get("document_id") +.retryWhen(RetryBuilder + .anyOf(BackpressureException.class, RequestCancelledException.class) + .delay(Delay.fixed(100, TimeUnit.MILLISECONDS)) + .max(20) + .build() +) +.timeout(2, TimeUnit.SECONDS) +.toBlocking() +.single();This example can be modified to produce the variationbelow, where different retry strategies are used for different exception types. A request cancellation might be retried immediately (1ms), whereas backpressure signalling overload may need a more gradual back off, with an exponential strategy:bucket.async() +.get("document_id") +.retryWhen(RetryBuilder + .anyOf(BackpressureException.class) + .delay(Delay.exponential(TimeUnit.MILLISECONDS, 1000, 1)) + .max(Integer.MAX_VALUE) + .build() +) +.retryWhen(RetryBuilder + .anyOf(RequestCancelledException.class) + .delay(Delay.fixed(1, TimeUnit.MILLISECONDS)) + .max(2) + .build() +) +.timeout(2, TimeUnit.SECONDS) +.toBlocking() +.single();Note the following values (and their defaults), which we override above:
      +
    • max attempts => 1
      • +
    • default retry delay => constant 1ms
      • +
    • anyOf errors => all errors are retried by default
    +
    +     +
    diff --git a/content/sdk/java/n1ql-queries-with-sdk.dita b/content/sdk/java/n1ql-queries-with-sdk.dita index edb59a5201..ddf4147c77 100644 --- a/content/sdk/java/n1ql-queries-with-sdk.dita +++ b/content/sdk/java/n1ql-queries-with-sdk.dita @@ -174,6 +174,35 @@ N1qlQuery query = N1qlQuery.simple("select count(*) from `mybucket`", params); +
    Long running Streaming Query +

    For a long-running N1QL query, you need a way of streaming the rows and correctly handling errors. Use this helper function:public static Observable<AsyncN1qlQueryRow> query(final Bucket bucket, final N1qlQuery query) { + return bucket.async() + .query(query).flatMap(new Func1<AsyncN1qlQueryResult, Observable<AsyncN1qlQueryRow>>() { + @Override + public Observable<AsyncN1qlQueryRow> call(AsyncN1qlQueryResult result) { + return Observable + .merge(result.rows(), result.errors()) + .map(new Func1<Object, AsyncN1qlQueryRow>() { + @Override + public AsyncN1qlQueryRow call(Object o) { + if (o instanceof AsyncN1qlQueryRow) { + return (AsyncN1qlQueryRow) o; + } else { + // The actual exception type and format could be improved further + throw new CouchbaseException(o.toString()); + } + } + }); + } + }); +}

    +

    To iterate over the rows, with blocking, call the helper function like this:Observable<AsyncN1qlQueryRow> rows = query(bucket, N1qlQuery.simple("select count(*) as cnt from `travel-sample`")); + +for (AsyncN1qlQueryRow row : rows.toBlocking().toIterable()) { + System.err.println(row); +}

    +

    In this example, this will print {"cnt":31591} to the console.

    +
    Conditionals, Case Expressions Mini DSL diff --git a/content/sdk/java/sample-app-backend.dita b/content/sdk/java/sample-app-backend.dita index 5bff9d843d..08d777055c 100644 --- a/content/sdk/java/sample-app-backend.dita +++ b/content/sdk/java/sample-app-backend.dita @@ -8,7 +8,7 @@

    The full source code for the tutorial is available on GitHub couchbaselabs/try-cb-Java. The primary focus of the tutorial is to explain the function and theory behind the Couchbase Java client and how it works together with Couchbase Server, and especially new features in versions 4.0/4.5 like N1QL, FTS and sub-document. It makes use of the travel-sample data set. The code that generates the web application is provided with the source code, @@ -65,14 +65,8 @@ method bodies and try to take it from there.

    This tutorial focuses on querying through N1QL and FTS rather than - mapreduce views. If you want information about using views, see the following - resources:
      -
    • General Information about views: -
      • -
    • Querying views:
      • -
    + views. If you want information about using views, see + Views.
    Walking Through the API @@ -390,7 +384,7 @@ private static List> extractResultOrThrow(N1qlQueryResult re
    Indexing the Data: N1QL & GSI

    Where: src/main/java/trycb/utils/StartupPreparations.java

    Goals: Use the Index DSL to make sure data is indexed for N1QL to query it.

    @@ -404,7 +398,7 @@ private static List> extractResultOrThrow(N1qlQueryResult re on(...) which Bucket.

    You can also create secondary indexes on specific fields of the JSON, for better performance:

    - +

    In this case, give a name to your index, specify the target bucket AND the field(s) in the JSON to index.

    diff --git a/content/sdk/java/scan-consistency-examples.dita b/content/sdk/java/scan-consistency-examples.dita new file mode 100644 index 0000000000..0a8aa42f74 --- /dev/null +++ b/content/sdk/java/scan-consistency-examples.dita @@ -0,0 +1,55 @@ + + + + N1QL Scan Consistency Using the Java SDK with Couchbase Server + Using Scan Consistency + You can balance performance against consistency in N1QL queries via the Couchbase Java client and the STATEMENT_PLUS (AtPlus) option. + +
    +

    N1QL provides a couple of options for scan consistency, represented in Java by the ScanConsistency enum:NOT_BOUNDED +REQUEST_PLUS +STATEMENT_PLUS + "Not bounded" being the default, the query engine will grab and return what is in the index right now. It is the fastest option, but not suitable for "read your own writes" since they may not yet be indexed.

    +

    At the other end of the scale there is "request plus" which will — when a query is performed — wait until all of the indexes used have caught up to their highest internal sequence numbers. + This will make sure that you read your own writes, but it might take some time if there is a write-heavy workload. Also sometimes you are only interested in the last mutation(s) you performed, so waiting for everything else in the index to be updated can mean unnecessary latency.

    +

    This is where "statement plus" comes in, also named "at plus". + This option will send some information previously retrieved at mutation time to the query engine which will then make sure to only wait for (at least) those mutations to have been indexed.

    +

    If statement_plus is used, mutation tokens need to be enabled in the environment:CouchbaseEnvironment env = DefaultCouchbaseEnvironment + .builder() + .mutationTokensEnabled(true) + .build();

    +

    There is a downside: from now on every mutation will return a couple more bytes of metadata, so there is slightly more overhead over the network. + Note though that this mechanism is also used for better handling of durability requirements (ReplicateTo/PersistTo), so the user might have it already enabled in their environment.

    +

    Now when a mutation is performed the MutationToken is part of the response. + Thus:JsonDocument written = bucket.upsert(JsonDocument.create("mydoc", JsonObject.empty())); +System.err.println(written.mutationToken());

    +

    Which will print:mt{vbID=55, vbUUID=166779084590420, seqno=488, bucket=travel-sample}

    +

    This information can then be passed in at query time:bucket.query( + N1qlQuery.simple("select count(*) as cnt from `travel-sample`", + N1qlParams.build().consistentWith(written)) +);

    +

    If you enable trace logging, you can see that on the wire the appropriate token vector is passed along to the query engine, which will handle the rest:[cb-io-1-2] 2018-07-09 13:28:01 TRACE LoggingHandler:94 - [id: 0x439d3980, L:/127.0.0.1:53732 - R:localhost/127.0.0.1:8093] WRITE: 231B + +-------------------------------------------------+ + | 0 1 2 3 4 5 6 7 8 9 a b c d e f | ++--------+-------------------------------------------------+----------------+ +|00000000| 7b 22 73 74 61 74 65 6d 65 6e 74 22 3a 22 73 65 |{"statement":"se| +|00000010| 6c 65 63 74 20 63 6f 75 6e 74 28 2a 29 20 61 73 |lect count(*) as| +|00000020| 20 63 6e 74 20 66 72 6f 6d 20 60 74 72 61 76 65 | cnt from `trave| +|00000030| 6c 2d 73 61 6d 70 6c 65 60 22 2c 22 63 6c 69 65 |l-sample`","clie| +|00000040| 6e 74 5f 63 6f 6e 74 65 78 74 5f 69 64 22 3a 22 |nt_context_id":"| +|00000050| 66 32 32 64 31 36 36 61 2d 35 38 30 35 2d 34 30 |f22d166a-5805-40| +|00000060| 62 36 2d 62 31 65 63 2d 66 62 62 35 61 66 36 33 |b6-b1ec-fbb5af63| +|00000070| 30 37 37 63 22 2c 22 73 63 61 6e 5f 76 65 63 74 |077c","scan_vect| <---- +|00000080| 6f 72 73 22 3a 7b 22 74 72 61 76 65 6c 2d 73 61 |ors":{"travel-sa| <---- +|00000090| 6d 70 6c 65 22 3a 7b 22 35 35 22 3a 5b 34 38 38 |mple":{"55":[488| <---- +|000000a0| 2c 22 31 36 36 37 37 39 30 38 34 35 39 30 34 32 |,"16677908459042| <---- +|000000b0| 30 22 5d 7d 7d 2c 22 74 69 6d 65 6f 75 74 22 3a |0"]}},"timeout":| <---- +|000000c0| 22 37 35 30 30 30 6d 73 22 2c 22 73 63 61 6e 5f |"75000ms","scan_| +|000000d0| 63 6f 6e 73 69 73 74 65 6e 63 79 22 3a 22 61 74 |consistency":"at| +|000000e0| 5f 70 6c 75 73 22 7d |_plus"} | ++--------+-------------------------------------------------+----------------+

    +
    + +     +     diff --git a/content/sdk/java/sdk-authentication-overview.dita b/content/sdk/java/sdk-authentication-overview.dita index 3667c2bfc7..f85650637f 100644 --- a/content/sdk/java/sdk-authentication-overview.dita +++ b/content/sdk/java/sdk-authentication-overview.dita @@ -20,6 +20,37 @@
    +
    + + + Legacy Connection Code + + +

    + These code snippets will enable you to use older, non-RBAC SDK clients with Couchbase Server 5.0 (and beyond). +

    + +
    Upgraded Couchbase Server 5.0 with passwordless buckets set up in an older version of Couchabse Server:
    +
    If a cluster with passwordless buckets is upgraded to Couchbase Server 5.0, the corresponding RBAC user entries are generated with username equal to the bucket name and with role set to "Full Bucket Access". + This process enables older SDK versions to access the buckets without creating the "Authentication Object". + The following snippet for a Java SDK with version < 2.4.4 will successfully return the bucket object as there is a user which has the username "beer-sample" (the same as the bucket name). +Cluster cluster = CouchbaseCluster.create(<hostname>); +Bucket bucket = cluster.openBucket("beer-sample"); +
    +     + + +
    Connecting to buckets created in Couchbase Server 5.0 from pre-RBAC SDK Verison:
    +
    To connect to buckets created in Couchbase Server 5.0 itself, using older client versions, you need to manually create a user with username equal the bucket name, and with role set to "Bucket Full Access". + The client can then connect to the bucket using the following sample snippet: +Cluster cluster = CouchbaseCluster.create(<hostname>); +Bucket bucket = cluster.openBucket(<bucket-name>, <password for the newly created user>); +
    +     +
    +

    +
    +
    @@ -85,8 +116,7 @@ <li> <i>Roles</i> required for resource-access, and the privileges they entail, see - <xref href="../../security/concepts-rba-for-apps.dita" scope="local" format="dita">RBAC - for Applications</xref>. + <xref href="../../security/security-roles.dita" scope="local" format="dita">Roles</xref>. <p> @@ -107,7 +137,8 @@

    - Couchbase Server supports the use of X.509 certificates to authenticate clients. This allows + Couchbase Server supports the use of X.509 certificates to authenticate clients (only available in + the Enterprise Edition, not the Community Edition). This allows authenticated users to access specific resources by means of the data service (no other service is currently available to clients through this form of authentication).

    @@ -127,7 +158,7 @@

    Note that this means that the explicit authentication process otherwise required by Couchbase Role-Based Access Control — whereby, in Java, username and password are passed by means of the - authenticate method on the Cluster object — is not needed. + authenticate method on the Cluster object — must not be used (instead the CertAuthenticator is passed in). (See Start Using the SDK for an example of such standard authentication.) @@ -148,14 +179,14 @@

    -
    +
    Authenticating a Java Client by Certificate

    To authenticate with Couchbase Server by means of a client certificate, a Java application - must have access to an appropriate keystore. A + must have access to an appropriate keystore or truststore. A procedure for the creation of such a keystore is provided in X.509 for TLS. The following Java code assumes that this procedure has been followed. The procedure's resulting @@ -170,14 +201,15 @@ import com.couchbase.client.java.Bucket; import com.couchbase.client.java.Cluster; import com.couchbase.client.java.CouchbaseCluster; +import com.couchbase.client.java.auth.CertAuthenticator; import com.couchbase.client.java.document.JsonDocument; import com.couchbase.client.java.env.CouchbaseEnvironment; import com.couchbase.client.java.env.DefaultCouchbaseEnvironment; -public class CertificateTest -{ - public static void main(String [] args) - { +public class CertificateTest { + + public static void main(String... args) { + CouchbaseEnvironment env = DefaultCouchbaseEnvironment .builder() .sslEnabled(true) @@ -187,18 +219,19 @@ public class CertificateTest .build(); Cluster userCluster = CouchbaseCluster.create(env, "10.143.173.101"); - + userCluster.authenticate(CertAuthenticator.INSTANCE); Bucket travelSample = userCluster.openBucket("travel-sample"); JsonDocument returnedAirline10doc = travelSample.get("airline_10"); System.out.println("Found: " + returnedAirline10doc); } + }

    As this example shows, certificate-based authentication is enabled by means of - the CouchbaseEnvironment class. (See + the CouchbaseEnvironment class as well as passing in the CertAuthenticator. (See Client Settings for details). The certificate-related options are as follows:

    diff --git a/content/sdk/java/sdk-user-management-overview.dita b/content/sdk/java/sdk-user-management-overview.dita index 9691972401..4c6258084d 100644 --- a/content/sdk/java/sdk-user-management-overview.dita +++ b/content/sdk/java/sdk-user-management-overview.dita @@ -66,8 +66,7 @@
  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    diff --git a/content/sdk/java/start-using-sdk.dita b/content/sdk/java/start-using-sdk.dita index 76bd30b57e..e75eb7a747 100644 --- a/content/sdk/java/start-using-sdk.dita +++ b/content/sdk/java/start-using-sdk.dita @@ -25,7 +25,7 @@ com.couchbase.client java-client - 2.5.3 + 2.6.2 ]]> @@ -75,7 +75,7 @@ public class Example { // Perform a N1QL Query N1qlQueryResult result = bucket.query( - N1qlQuery.parameterized("SELECT name FROM bucketname WHERE $1 IN interests", + N1qlQuery.parameterized("SELECT name FROM `bucketname` WHERE $1 IN interests", JsonArray.from("African Swallows")) ); @@ -165,7 +165,8 @@ cluster.disconnect();]]>

    API Reference -

    Each stable Java SDK version has the Javadocs API reference published. The reference for the latest version can be found here.

    +

    Each stable Java SDK version has the Javadocs API reference published. The reference for the latest version can + be found here.

    Contributing diff --git a/content/sdk/java/threshold-logging.dita b/content/sdk/java/threshold-logging.dita new file mode 100644 index 0000000000..bb15d66d69 --- /dev/null +++ b/content/sdk/java/threshold-logging.dita @@ -0,0 +1,47 @@ + + + + Threshold Logging Tracing through the SDK + Threshold Logging + + + +
    +
    + +
    +
    + +
    +
    + + +
    + + Threshold Logging in Java + + +

    + Response Time Observability is implemented as Threshold Logging in Java SDK from release 2.6.0.

    +
    + + +
    + + Customizing + +

    + You can customise Tracer settings through the CouchbaseEnvironment.Builder. Here + is an example on how to customize our default tracer to reduce the time interval when the information + gets logged:Tracer tracer = ThresholdLogTracer.create(ThresholdLogReporter.builder() + .logInterval(10, TimeUnit.SECONDS) // log every 10 seconds + .build()); + +CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder() + .tracer(tracer) + .build(); + Follow along with our example of , to discover how to use tracing in the Java SDK for Response Time Observability. +

    +
    + +     \ No newline at end of file diff --git a/content/sdk/java/tracing-from-the-sdk.dita b/content/sdk/java/tracing-from-the-sdk.dita new file mode 100644 index 0000000000..81385b0dac --- /dev/null +++ b/content/sdk/java/tracing-from-the-sdk.dita @@ -0,0 +1,151 @@ + + + + Tracing from the Java SDK with Couchbase Server + Tracing from the SDK + Use tracing from the SDK to discover timeouts and bottlenecks across the network and cluster. + +

    For our example, we will customize the threshold logging tracer settings with CouchbaseEnvironment.Builder. By default it + will log every minute (if something is found) and only sample the top 10 slowest operations. The default threshold for key/value operation is 500 milliseconds. We shall + set them so low that almost everything gets logged - not something you should do in production!Tracer tracer = ThresholdLogTracer.create(ThresholdLogReporter.builder() + .kvThreshold(1, TimeUnit.MICROSECONDS) // 1 microsecond + .logInterval(1, TimeUnit.SECONDS) // log every second + .sampleSize(Integer.MAX_VALUE) + .pretty(true) // pretty print the json output in the logs + .build());This will set our threshold for key/value operations to one microsecond, log the found operations every second, and set the sample size to a very large value so everything will be logged.

    + +

    With these configs in place we are ready to run some operations. Below, we read some documentss from the travel-sample bucket and, if found, write them back with an upsert - giving us both read and write operations to log.// Connect +CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder() + .tracer(tracer) + .build(); +Cluster cluster = CouchbaseCluster.create(env, "127.0.0.1"); +cluster.authenticate("username", "password"); + + +Bucket bucket = cluster.openBucket("travel-sample"); + +// Load a couple of docs and write them back +for(int i = 0; i < 5; i++) { + JsonDocument doc = bucket.get("airline_1" + i); + if (doc != null) { + bucket.upsert(doc); + } +} + +Thread.sleep(TimeUnit.MINUTES.toMillis(1));

    + +

    Run the code, and you will see something like the following in the logs:Apr 04, 2018 9:42:57 AM com.couchbase.client.core.tracing.ThresholdLogReporter logOverThreshold +WARNING: Operations over threshold: [ { + "top" : [ { + "server_us" : 8, + "last_local_id" : "41837B87B9B1C5D1/000000004746B9AA", + "last_local_address" : "127.0.0.1:55011", + "last_operation_id" : "get:0x6", + "last_dispatch_us" : 315, + "last_remote_address" : "127.0.0.1:11210", + "total_us" : 576 + }, { + "server_us" : 8, + "last_local_id" : "41837B87B9B1C5D1/000000004746B9AA", + "last_local_address" : "127.0.0.1:55011", + "last_operation_id" : "get:0x5", + "last_dispatch_us" : 319, + "last_remote_address" : "127.0.0.1:11210", + "total_us" : 599 + }, { + "server_us" : 8, + "last_local_id" : "41837B87B9B1C5D1/000000004746B9AA", + "last_local_address" : "127.0.0.1:55011", + "last_operation_id" : "get:0x4", + "last_dispatch_us" : 332, + "last_remote_address" : "127.0.0.1:11210", + "total_us" : 632 + }, { + "server_us" : 11, + "last_local_id" : "41837B87B9B1C5D1/000000004746B9AA", + "last_local_address" : "127.0.0.1:55011", + "last_operation_id" : "get:0x3", + "last_dispatch_us" : 392, + "last_remote_address" : "127.0.0.1:11210", + "total_us" : 762 + }, { + "server_us" : 23, + "last_local_id" : "41837B87B9B1C5D1/000000004746B9AA", + "last_local_address" : "127.0.0.1:55011", + "last_operation_id" : "get:0x1", + "decode_us" : 9579, + "last_dispatch_us" : 947, + "last_remote_address" : "127.0.0.1:11210", + "total_us" : 16533 + }, { + "server_us" : 56, + "encode_us" : 12296, + "last_local_id" : "41837B87B9B1C5D1/000000004746B9AA", + "last_local_address" : "127.0.0.1:55011", + "last_operation_id" : "upsert:0x2", + "last_dispatch_us" : 1280, + "last_remote_address" : "127.0.0.1:11210", + "total_us" : 20935 + } ], + "service" : "kv", + "count" : 6 +} ]For each service (only kv-based on this workload), the threshold log reporter will show you the total number of recorded ops (through count), and give you the top slowest ops sorted by their latency. Since only airline_10 exists in the bucket you will see five document fetches but only one mutation.

    + + +
    + + Output fields in detail. + +

    Let's highlight a single operation, and explain each field in a little more detail:{ + "server_us" : 23, + "last_local_id" : "41837B87B9B1C5D1/000000004746B9AA", + "last_local_address" : "127.0.0.1:55011", + "last_operation_id" : "get:0x1", + "decode_us" : 1203, + "last_dispatch_us" : 947, + "last_remote_address" : "127.0.0.1:11210", + "total_us" : 1525 +}

    +

    This tells us the following:

      +
    • total_us: The total time it took to perform the full operation: here around 1.5 milliseconds.
      • +
    • server_us: The server reported that its work performed took 23 microseconds (this does not include network time or time in the buffer before picked up at the cluster).
      • +
    • decode_us: Decoding the response took the client 1.2 milliseconds.
      • +
    • last_dispatch_us: The time when the client sent the request and got the response took around 1 millisecond.
      • +
    • last_local_address: The local socket used for this operation.
      • +
    • last_remote_address: The remote socket on the server used for this operation. Useful to figure out which node is affected.
      • +
    • last_operation_id: A combination of type of operation and id (in this case the opaque value), useful for diagnosing and troubleshooting in combination with the last_local_id.
      • +
    • last_local_id: With Server 5.5 and later, this id is negotiated with the server and can be used to correlate logging information on both sides in a simpler fashion.

    +

    You can see that if the thresholds are set the right way based on production requirements, without much effort slow operations can be logged and pinpointed more easily than before.

    +
    + + +
    + + Timeout Visibility. + +

    Previously, when an operation takes longer than the timeout specified allows, a TimeoutException is thrown. It usually looks like this:.Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.TimeoutException: {"b":"travel-sample","r":"127.0.0.1:11210","s":"kv","c":"30893646E8E78A3E/FFFFFFFFDE1ED905","t":10000,"i":"0x1","l":"127.0.0.1:55821"} + at rx.exceptions.Exceptions.propagate(Exceptions.java:57) + at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463) + at rx.observables.BlockingObservable.singleOrDefault(BlockingObservable.java:372) + at com.couchbase.client.java.CouchbaseBucket.get(CouchbaseBucket.java:131) + at Main.main(Main.java:53) +Caused by: java.util.concurrent.TimeoutException: {"b":"travel-sample","r":"127.0.0.1:11210","s":"kv","c":"30893646E8E78A3E/FFFFFFFFDE1ED905","t":10000,"i":"0x1","l":"127.0.0.1:55821"} + at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:131) + at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:127) + at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) + at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166) + at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191) + at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) + at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) + at java.util.concurrent.FutureTask.run(FutureTask.java:266) + at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) + at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) + at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) + at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) + at java.lang.Thread.run(Thread.java:745)Now the timeout itself provides you valuable information like the local and remote sockets, and the operation id, as well as the timeout set and the local ID used for troubleshooting. You can take this information and correlate it to the top slow operations in the threshold log.

    + +

    The TimeoutException now provides you more information into what went wrong and then you can go look at the log to figure out why it was slow. +

    +
    + +     \ No newline at end of file diff --git a/content/sdk/nodejs.ditamap b/content/sdk/nodejs.ditamap index ba460575e3..2a8d32048e 100644 --- a/content/sdk/nodejs.ditamap +++ b/content/sdk/nodejs.ditamap @@ -19,6 +19,9 @@ + + + @@ -40,6 +43,7 @@ + @@ -51,5 +55,7 @@ + + diff --git a/content/sdk/nodejs/async-programming.dita b/content/sdk/nodejs/async-programming.dita index 7cdfc487c7..062b34eeda 100644 --- a/content/sdk/nodejs/async-programming.dita +++ b/content/sdk/nodejs/async-programming.dita @@ -5,8 +5,8 @@ Asynchronous Progamming Using the Node.js SDK with Couchbase Server Asynchronous Programming -

    The Node.js SDK supports asynchronous programming implicitly through it's use of the typical - Node.js callback pattern coupled with having our underlying networking library tightly coupled +

    The Node.js SDK supports asynchronous programming implicitly through its use of the typical + Node.js callback pattern, coupled with having our underlying networking library tightly coupled to the Node.js event loop. Performing many operations asynchronously garners a huge performance benefit compared to performing operations in a serial manner. The same bucket can have hundreds of outstanding operations at any one time with minimal loss in performance diff --git a/content/sdk/nodejs/compatibility-versions-features.dita b/content/sdk/nodejs/compatibility-versions-features.dita index b1f71b75ef..d0e0dd2cfc 100644 --- a/content/sdk/nodejs/compatibility-versions-features.dita +++ b/content/sdk/nodejs/compatibility-versions-features.dita @@ -14,7 +14,7 @@

  • ✖ Unsupported: This combination is not tested, and is not within the scope of technical support if you have purchased a support agreement.
  • ◎ Compatible: This combination has been tested previously, and - should be compatible. This combination is not supported or recommended by + should be compatible. This combination is not recommended by our technical support organization. It is best to upgrade either the SDK or the Couchbase version you are using.
  • ✔ Supported:This combination is subject to ongoing quality @@ -29,70 +29,30 @@ - CB/SDK SDK 2.0 - SDK 2.1 - SDK 2.2 - SDK 2.3 + SDK 2.1 - SDK 2.3 + SDK 2.4 - - CB 2.5 - ◎ - ◎ - ◎ - ◎ - - - CB 3.0 - ◎ - ◎ - ◎ - ◎ - CB 3.1 ◎ ✔ ✔ - ✔ - - - CB 4.0 - ✔ - ✔ - ✔ - ✔ - - - CB 4.1 - ✔ - ✔ - ✔ - ✔ - - - CB 4.5 - ✔ - ✔ - ✔ - ✔ - CB 4.6 - ✔ + CB 4.0 - 4.6 ✔ ✔ ✔ - CB 5.0 - ✔ - ✔ + CB 5.0 - 5.1 + ✖ ✔ ✔ diff --git a/content/sdk/nodejs/encrypting-using-sdk.dita b/content/sdk/nodejs/encrypting-using-sdk.dita new file mode 100644 index 0000000000..c5301c58e2 --- /dev/null +++ b/content/sdk/nodejs/encrypting-using-sdk.dita @@ -0,0 +1,51 @@ + + + + Field Level Encryption from the Java SDK + Field Level Encryption is available in Couchbase Data Platform 5.5, from Node.js SDK version 2.5.0 + +
    + Packaging +

    The Couchbase Node.js SDK uses the node-couchbase-encryption library to provide support for encryption and decryption of JSON fields.

    + +

    The Couchbase Node.js Field Level Encryption (FLE) uses a list of fields mapped to crypto providers to define which field(s) to apply encryption to, and which algorithm to use. You must also configure a key store to use with your providers. In this example we use the “InsecureKeyStore” in-memory store for development and testing - don't use this one in production!var publicKey = '!mysecretkey#9^5usdk39d&dlf)03sL'; +var signingKey = 'myauthpassword'; + +var keyStore = new cbfieldcrypt.InsecureKeyStore(); +keyStore.addKey('publickey', publicKey); +keyStore.addKey('mysecret', signingKey); + +var personCryptFields = { + password: new cbfieldcrypt.AesCryptoProvider(keyStore, 'publickey', 'mysecret') +};

    + +
    + Encryption +

    To apply encryption to an object you are writing to Couchbase Server, use the encrypt function with your provider map:var encryptedTeddy = cbfieldcrypt.encryptFields(teddy, personCryptFields); + +bucket.upsert('person::1', encryptedTeddy, function(err, res) { + if (err) { + throw err; + } + + // ... +});

    + + +
    + Decrypting +

    To remove encryption from an object which was previously encrypted and stored in Couchbase, use the decrypt function, again with your provider map:bucket.get('person::1', function(err, res) { + if (err) { + throw err; + } + + var encryptedData = res.value; + var decryptedData = + cbfieldcrypt.decryptFields(encryptedData, personCryptFields); + + // ... +});

    + +     +     \ No newline at end of file diff --git a/content/sdk/nodejs/full-text-searching-with-sdk.dita b/content/sdk/nodejs/full-text-searching-with-sdk.dita index 19f57edd84..290aa0b5d1 100644 --- a/content/sdk/nodejs/full-text-searching-with-sdk.dita +++ b/content/sdk/nodejs/full-text-searching-with-sdk.dita @@ -6,7 +6,10 @@ Using the Node.js SDK, you can use the Couchbase Full Text Search service (FTS) to create queryable full-text indexes in Couchbase Server.

    Couchbase offers Full-text search support, allowing you to search for documents that contain - certain words or phrases. In the Node.js SDK you can search full-text indexes by using the + certain words or phrases. A general introduction to FTS, with pointers to detailed descriptions of its + principal features, is provided in + Full Text Search: Fundamentals. In + the Node.js SDK you can search full-text indexes by using the SearchQuery object and executing it with the Bucket.query() API.

    The following example shows how to send a simple Search query:

    @@ -51,10 +54,421 @@ bucket.query(query, function(err, res, meta) { Facet object using methods found in the SearchFacet class (i.e.: SearchFacet.term() for a TermFacet).

    var query = SearchQuery.new('travel-search', SearchQuery.term('office')); -query.addFacet('countries', SearchFacet.term('country', 5); +query.addFacet('countries', SearchFacet.term('country', 5)); bucket.query(query, function(err, res, meta) { console.log('Total Countries:', meta.facets['countries'].total); -}); +});
    • +
    Detailed Examples + +

    + The code example below demonstrates the Node.js SDK Full Text Search + API. The example assumes that Couchbase Server is running, and that + the username Administrator and + the password password provide authorization for performing the searches. + It also assumes that the travel-sample bucket + has been installed. + For information on creating users and managing roles, see + Authorization. + For information on installing sample buckets, see + Settings. +

    + +

    + The example also assumes the existence of three specific Full Text Indexes, defined on the + travel-sample bucket. These are: +

    + +
      +
    • + travel-sample-index-unstored: Uses only the default + settings. +
      • + +
    • + travel-sample-index-stored: Uses default settings, with one exception: + dynamic fields are stored, for the whole index. +
      • + +
    • + travel-sample-index-hotel-description: Indexes only the description + fields of hotel + documents, and disables the default type mapping. The index + has a custom analyzer named myUnicodeAnalyzer defined on it: the analyzer's + main characteristic is that it uses the unicode tokenizer. +
      • +
    + +

    + See + Creating Indexes + for details on how to create these indexes: they can be created interactively, by + means of the Couchbase Web Console; however, there may be greater efficiency + in using the Couchbase REST + API, as described + in the section + Index Creation with the REST API. + The JSON objects that constitute index-definitions (for + inclusion as bodies to the index-creation REST calls), are provided in + Demonstration Indexes. +

    + +

    + The example features the following Full Text Searches on the + travel-sample bucket, within Couchbase Server: +

    + +
      + +
    • + Simple Text Query on a single word, targeting an index with dynamic fields unstored. +
      • + +
    • + Simple Text Query on Non-Default Index, specifying an index + that consists only of content derived from a specific field from a + specific document-type. +
      • + +
    • + Simple Text Query on Stored Field, specifying the field to be searched; targeting an index with dynamic + fields stored, to ensure that field-content is included in the return object. +
      • + +
    • + Match Query with Facet, showing how query-results can be displayed + either by row or by hits; + and demonstrating use of a facet, which provides aggregation-data. +
      • + +
    • + DocId Query, showing results of a query on two document IDs. +
      • + +
    • + Unanalyzed Term Query with Fuzziness Level of 0, demonstrating how to query on a term with no analysis. + Zero fuzziness is specified, to ensure + that matches are exact. +
      • + +
    • + Unanalyzed Term Query with Fuzziness Level of 2, which is almost identical to the immediately preceding query; but which this time + specifies a fuzziness factor of 2, allowing partial matches + to be made. The + output from this query can be compared to that of the one immediately preceding. +
      • + +
    • + Match Phrase Query, using Analysis, for searching on a phrase. +
      • + +
    • + Phrase Query, without Analysis, for searching on a phrase without analysis supported. +
      • + +
    • + Query String Query, showing how a query string is specified as search-input. +
      • + +
    • + Conjunction Query, whereby two separate queries are defined and then run as part of the search, + with only the matches returned by both included in the result-object. +
      • + + +
    • + Wild Card Query, whereby a wildcard is used in the string submitted for the search. +
      • + +
    • + Numeric Range Query, whereby minimum and maximum numbers are specified, and matches + within the range returned. +
      • + +
    • + Regexp Query, whereby a regular expression is submitted, to generate the conditions for + successful matches. +
      • + +
    + 'use strict'; + +var couchbase = require('couchbase'); + +var cluster = new couchbase.Cluster('couchbase://10.111.181.101'); +cluster.authenticate('Administrator', 'password'); +var travelSample = cluster.openBucket('travel-sample'); +var SearchQuery = couchbase.SearchQuery; +var SearchFacet = couchbase.SearchFacet; + +function simpleTextQuery(bucket, done) { + var indexName = 'travel-sample-index-unstored'; + var match = SearchQuery.match('swanky'); + var query = SearchQuery.new(indexName, match).limit(10); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Simple Text Query Error:', err); + done(); + return; + } + printResult('Simple Text Query:', res); + done(); + }); +} + +function simpleTextQueryOnStoredField(bucket, done) { + var indexName = 'travel-sample-index-stored'; + var match = SearchQuery.match('MDG').field('destinationairport'); + var query = SearchQuery.new(indexName, match).limit(10).highlight(); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Simple Text Query on Stored Field Error:', err); + done(); + return; + } + printResult('Simple Text Query on Stored Field:', res); + done(); + }); +} + +function simpleTextQueryOnNonDefaultIndex(bucket, done) { + var indexName = 'travel-sample-index-hotel-description'; + var match = SearchQuery.match('swanky'); + var query = SearchQuery.new(indexName, match).limit(10); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Simple Text Query on Non-Default Index Error:', err); + done(); + return; + } + printResult('Simple Text Query on Non-Default Index:', res); + done(); + }); +} + +function textQueryOnStoredFieldWithFacet(bucket, done) { + var indexName = 'travel-sample-index-stored'; + var match = SearchQuery.match('La Rue Saint Denis!!').field('reviews.content'); + var query = SearchQuery.new(indexName, match).limit(10).highlight() + .addFacet('Countries Referenced', SearchFacet.term('country', 5)); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Match Query with Facet Error:', err); + done(); + return; + } + console.log(); + console.log('Match Query with Facet, Result by hits:'); + for (var i = 0; i < res.length; ++i) { + console.log(JSON.stringify(res[i])); + } + + console.log(); + console.log('Match Query with Facet, Result by facets:'); + for (var facet in meta.facets) { + console.log(facet, ':', JSON.stringify(meta.facets[facet])); + } + done(); + }); +} + +function docIdQueryMethod(bucket, done) { + var indexName = 'travel-sample-index-unstored'; + var query = SearchQuery.new(indexName, SearchQuery.docIds('hotel_26223', 'hotel_28960')); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('DocId Query Error:', err); + done(); + return; + } + printResult('DocId Query:', res); + done(); + }); +} + +function unAnalyzedTermQuery(bucket, fuzzinessLevel, done) { + var indexName = 'travel-sample-index-stored'; + var term = SearchQuery.term('sushi').field('reviews.content') + .fuzziness(fuzzinessLevel); + var query = SearchQuery.new(indexName, term).limit(50).highlight(); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Unanalyzed Term Query with Fuzziness Level of', fuzzinessLevel, 'Error:', err); + done(); + return; + } + printResult('Unanalyzed Term Query with Fuzziness Level of ' + fuzzinessLevel + ':', res); + done(); + }); +} + +function matchPhraseQueryOnStoredField(bucket, done) { + var indexName = 'travel-sample-index-stored'; + var match = SearchQuery.matchPhrase('Eiffel Tower') + .field('description'); + + var query = SearchQuery.new(indexName, match).limit(10).highlight(); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Match Phrase Query, using Analysis Error:', err); + done(); + return; + } + printResult('Match Phrase Query, using Analysis:', res); + done(); + }); +} + +function unAnalyzedPhraseQuery(bucket, done) { + var indexName = 'travel-sample-index-stored'; + var phrase = SearchQuery.phrase(['dorm', 'rooms']) + .field('description'); + + var query = SearchQuery.new(indexName, phrase).limit(10).highlight(); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Phrase Query, without Analysis Error:', err); + done(); + return; + } + printResult('Phrase Query, without Analysis:', res); + done(); + }); +} + +function conjunctionQueryMethod(bucket, done) { + var indexName = 'travel-sample-index-stored'; + var firstQuery = SearchQuery.match('La Rue Saint Denis!!') + .field('reviews.content'); + var secondQuery = SearchQuery.match('boutique') + .field('description'); + + var conjunctionQuery = SearchQuery.conjuncts(firstQuery, secondQuery); + + var query = SearchQuery.new(indexName, conjunctionQuery).limit(10).highlight(); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Conjunction Query Error:', err); + done(); + return; + } + printResult('Conjunction Query:', res); + done(); + }); +} + +function queryStringMethod(bucket, done) { + var indexName = 'travel-sample-index-unstored'; + var queryString = SearchQuery.queryString('description: Imperial'); + var query = SearchQuery.new(indexName, queryString).limit(10); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Query String Query Error:', err); + done(); + return; + } + printResult('Query String Query:', res); + done(); + }); +} + +function wildCardQueryMethod(bucket, done) { + var indexName = 'travel-sample-index-stored'; + var wildcard = SearchQuery.wildcard('bouti*ue').field('description'); + var query = SearchQuery.new(indexName, wildcard).limit(10).highlight(); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Wild Card Query Error:', err); + done(); + return; + } + printResult('Wild Card Query:', res); + done(); + }); +} + +function regexpQueryMethod(bucket, done) { + var indexName = 'travel-sample-index-stored'; + var reg = SearchQuery.regexp('[a-z]').field('description'); + var query = SearchQuery.new(indexName, reg).limit(10).highlight(); + bucket.query(query, function(err, res, meta) { + if (err) { + console.log(); + console.log('Regexp Query Error:', err); + done(); + return; + } + printResult('Regexp Query:', res); + done(); + }); +} + +function printResult(label, result) { + console.log(); + console.log('= = = = = = = = = = = = = = = = = = = = = = ='); + console.log('= = = = = = = = = = = = = = = = = = = = = = ='); + console.log(); + console.log(label); + console.log(); + + for (var i = 0; i < result.length; ++i) { + console.log(JSON.stringify(result[i])); + } +} + +function querySet1(bucket, done) { + simpleTextQuery(bucket, function() { + simpleTextQueryOnStoredField(bucket, function() { + simpleTextQueryOnNonDefaultIndex(bucket, function() { + textQueryOnStoredFieldWithFacet(bucket, done); + }); + }); + }); +} + +function querySet2(bucket, done) { + docIdQueryMethod(bucket, function() { + unAnalyzedTermQuery(bucket, 0, function() { + unAnalyzedTermQuery(bucket, 2, function() { + matchPhraseQueryOnStoredField(bucket, function() { + unAnalyzedPhraseQuery(bucket, done); + }); + }); + }); + }); +} + +function querySet3(bucket, done) { + conjunctionQueryMethod(bucket, function() { + queryStringMethod(bucket, function() { + wildCardQueryMethod(bucket, function() { + regexpQueryMethod(bucket, done); + }); + }); + }); +} + +querySet1(travelSample, function() { + querySet2(travelSample, function() { + querySet3(travelSample, function() { + travelSample.disconnect(); + process.exit(0); + }); + }); + }); + + +
    diff --git a/content/sdk/nodejs/managing-connections.dita b/content/sdk/nodejs/managing-connections.dita index 54da64c3e2..fe39cd0055 100644 --- a/content/sdk/nodejs/managing-connections.dita +++ b/content/sdk/nodejs/managing-connections.dita @@ -23,6 +23,13 @@ to:var couchbase = require('couchbase'); var cluster = new couchbase.Cluster('couchbase://10.0.0.1');

    • +
    + Authenticating +

    From Couchbase Server 5.0, you will need to authenticate the user, rather than against the + bucket, as part of Role-Based Access Control. You will need to + use cluster.authenticate:cluster.authenticate('USERNAME', 'PASSWORD'); +

    +
    Connecting to a Bucket

    To connect to a bucket, invoke the Cluster object's @@ -32,7 +39,7 @@ var cluster = new couchbase.Cluster('couchbase://10.0.0.1');

    openBucket also accepts a function as its final argument which reports any connection errors which may happen between the creation of the Bucket object and the actual - connection.var bucket = cluster.openBucket('protectedBucket', 's3cr3t', function(err) { + connection.var bucket = cluster.openBucket('protectedBucket', function(err) { if (err) { console.error('Got error: %j', err); } diff --git a/content/sdk/nodejs/n1ql-queries-with-sdk.dita b/content/sdk/nodejs/n1ql-queries-with-sdk.dita index 60c22d1725..664f8ae59a 100644 --- a/content/sdk/nodejs/n1ql-queries-with-sdk.dita +++ b/content/sdk/nodejs/n1ql-queries-with-sdk.dita @@ -69,5 +69,62 @@ Got a row Got a row All rows received. Metadata is {"requestID":"79914eb3-6204-4dc7-b9e6-b4680b6509a0","signature":{"*":"*"},"status":"success","metrics":{"elapsedTime":"8.631682ms","executionTime":"8.593926ms","resultCount":10,"resultSize":3002}}:

    - +
    + CAS via N1QL +

    For N1QL queries, the Node.js SDK accepts CAS values as strings. This is necessary as CAS values are + too large to be represented by numerical values in JavaScript. The following example illustrates + working with CAS values as strings.'use strict'; + +var couchbase = require('couchbase'); + +var cluster = new couchbase.Cluster('couchbase://localhost'); +cluster.authenticate('Administrator', 'password'); + + +var bucket = cluster.openBucket('travel-sample'); + +// Insert a document with a random x value +bucket.upsert('test-doc', {x: Math.round(Math.random()*10000000)}, function(err, res) { + if (err) { + throw err; + } + + var qs = 'SELECT t.*, TOSTRING(META().cas) AS `_cas` FROM `travel-sample` t WHERE META().id="test-doc"'; + var q = couchbase.N1qlQuery.fromString(qs); + q.consistency(couchbase.N1qlQuery.Consistency.REQUEST_PLUS); + bucket.query(q, function(err, rows) { + if (err) { + throw err; + } + + if (rows.length !== 1) { + throw new Error('unexpected number of rows'); + } + + console.log('Query Result:', rows[0]); + + var cas = rows[0]._cas; + var doc = rows[0]; + delete(doc._cas); + doc.y = doc.x; + + bucket.replace('test-doc', doc, {cas: cas}, function(err) { + if (err) { + throw err; + } + + bucket.get('test-doc', function(err, res) { + if (err) { + throw err; + } + + console.log('Updated:', res.value); + + process.exit(0); + }); + }); + }); +});

    +
    + diff --git a/content/sdk/nodejs/sample-app-backend.dita b/content/sdk/nodejs/sample-app-backend.dita index 7c2d4c0597..f961a2ef72 100644 --- a/content/sdk/nodejs/sample-app-backend.dita +++ b/content/sdk/nodejs/sample-app-backend.dita @@ -8,7 +8,7 @@

    The full source code for the tutorial is available on GitHub: http://github.com/couchbaselabs/try-cb-nodejs. The primary focus of the tutorial is to explain the function and theory behind the Couchbase Node.js client and how it works together with Couchbase Server, and especially new features in versions 4.0/4.5 like N1QL, FTS and sub-document. It makes use of the travel-sample data set. The code that generates the web application is provided with the source code.

    @@ -23,7 +23,7 @@

    To get set up for the tutorial proper, follow these steps:

    • git clone https://github.com/couchbaselabs/try-cb-nodejs.git or download the source.
    • Install necessary libraries: npm install.
      • diff --git a/content/sdk/nodejs/sdk-authentication-overview.dita b/content/sdk/nodejs/sdk-authentication-overview.dita index 20a9a7e5fb..f222fcd282 100644 --- a/content/sdk/nodejs/sdk-authentication-overview.dita +++ b/content/sdk/nodejs/sdk-authentication-overview.dita @@ -86,8 +86,7 @@
    • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

      @@ -101,7 +100,52 @@

    - +
    + + Certificate-Based Authentication + + +

    + Couchbase Server supports the use of X.509 certificates to authenticate clients (only available in + the Enterprise Edition, not the Community Edition). This allows + authenticated users to access specific resources by means of the data service (no other service + is currently available to clients through this form of authentication). +

    + +

    + The process relies on a certificate + authority, for the issuing of certificates that validate identities. A certificate includes + information such as the name of the entity it identifies, an expiration date, the name of the authority + that issued the certificate, and the digital signature of the authority. + A client attempting to access Couchbase Server can present a certificate to the server, allowing + the server to check the validity of the certificate. If the certificate is valid, the user + under whose identity the client is running, + and the roles assigned that user, are verified. If the assigned roles are appropriate for + the level of access requested to the specified resource, access is granted. +

    + +

    The Node.js SDK has supported certificate authentication since version 2.4.4. The client certificate can be generated with the shell script on + the C SDK Authentication page. + Connecting with the Node.js client, using a certificate, involves using cluster.authenticate + and passing the certificate and key paths in the connection string. When this authenticator is used for the authenticate method, it signals to the client to pick up any certificate/keystore for X509 auth:// Require Couchbase Module +var couchbase = require('couchbase'); + +// Setup Cluster Connection Object +var cluster = new couchbase.Cluster( + 'couchbases://127.0.0.1/' + + '?truststorepath=../etc/x509-cert/SSLCA/clientdir/trust.pem' + + '&certpath=../etc/x509-cert/SSLCA/clientdir/client.pem' + + '&keypath=../etc/x509-cert/SSLCA/clientdir/client.key'); + +// Authenticate with the cluster +cluster.authenticate(new couchbase.CertAuthenticator()); + +// Setup Bucket object to be reused within the code +var bucket = cluster.openBucket('travel-sample'); + +console.log('Example Successful - Exiting'); +process.exit(0);

    +
    diff --git a/content/sdk/nodejs/sdk-user-management-example.dita b/content/sdk/nodejs/sdk-user-management-example.dita index 7cb163ad0d..d4be0ab8c7 100644 --- a/content/sdk/nodejs/sdk-user-management-example.dita +++ b/content/sdk/nodejs/sdk-user-management-example.dita @@ -32,7 +32,10 @@

    - The following code-example demonstrates how the user-management APIs can be used. It assumes that Couchbase Server is + The following code-example demonstrates how the user-management APIs can be used. + It involves setting roles only + available in the Enterprise Edition. For Community Edition, adjust roles accordingly.

    +

    It assumes that Couchbase Server is established on localhost; that the Full Administrator username and password are Administrator and password respectively; and that the travel-sample bucket is @@ -66,7 +69,7 @@ cluster.authenticate('Administrator', 'password'); // Create a user and assign roles. console.log('Upserting new user'); -cluster.manager().upsertUser('cbtestuser', { +cluster.manager().upsertUser('localhost', 'cbtestuser', { password: 'cbtestuserpwd', roles: [ diff --git a/content/sdk/nodejs/sdk-user-management-overview.dita b/content/sdk/nodejs/sdk-user-management-overview.dita index 85af5fd416..67a8975209 100644 --- a/content/sdk/nodejs/sdk-user-management-overview.dita +++ b/content/sdk/nodejs/sdk-user-management-overview.dita @@ -66,8 +66,7 @@

  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    diff --git a/content/sdk/nodejs/start-using-sdk.dita b/content/sdk/nodejs/start-using-sdk.dita index 288e16c5e9..646e89275c 100644 --- a/content/sdk/nodejs/start-using-sdk.dita +++ b/content/sdk/nodejs/start-using-sdk.dita @@ -11,9 +11,10 @@

    Installing the SDK

    To install the SDK, simply use npm. On non-windows platforms you will also need to install node-gyp:

    npm install couchbase

    Some features such as SSL support require using a - manually installed version of the C SDK.

    + scope="external">node-gyp:

    npm install couchbase

    On + the Windows platform using Node.js versions prior to 6.0, manual installation of OpenSSL and + the C SDK (libcouchbase) is required. It is recommended + to use Node.js 6.0 and later.

    Information on new features, fixes, known issues as well as information on how to install older release versions is var couchbase = require('couchbase') var cluster = new couchbase.Cluster('couchbase://localhost/'); -cluster.authenticate(‘USERNAME’, ‘PASSWORD’); +cluster.authenticate('USERNAME', 'PASSWORD'); var bucket = cluster.openBucket('bucketname'); var N1qlQuery = couchbase.N1qlQuery; @@ -61,7 +62,7 @@ bucket.manager().createPrimaryIndex(function() { Additional Resources

    • An API reference is generated for each release and can be found at
    • The source code repsitory is at + + + Threshold Logging Tracing through the SDK + Threshold Logging + + + +
      +
      + +
      +
      + +
      +
      + + +
      + + Threshold Logging in the Node.js SDK + + +

      + Response Time Observability is implemented as Threshold Logging in the Node.js SDK from release 2.5.0.

      + + + Threshold Logging Tracer Properties + + + + Setting name + Description + Default Value + + + + + enable_tracing + Boolean used to determine tracing is enabled. Defaults to using the + ThesholdLoggingTracer if enabled. When false a Noop or similar tracing + implementation should be used instead. Also when enabled the SDK will try and retrieve + duration metrics from the server for KV operations. + true + + + tracing_threshold_queue_flush_interval + The interval between executions that process the collected operation spans. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 10.0 seconds (or 10000000 microseconds) + + + tracing_threshold_queue_size + The maximum number of items to keep in internal queue per service. + 128 + + + tracing_threshold_kv + The KV operation operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 0.5 second (500000 microseconds) + + + tracing_threshold_view + The View query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_n1ql + The N1QL query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_fts + The FTS query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_analytics + The Analytics query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_orphaned_queue_flush_interval + The interval between executions that processes the collected operation spans. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 10.0 seconds (or 10000000 microseconds) + + + tracing_orphaned_queue_size + The maximum number of items to keep in internal queue per service. + 128 + + + +
      +
      + + +
      + + Customizing + +

      + You can customise Tracer settings through the connection string, or using lcb_cntl. Here + is an example on how to customize our default tracer to reduce the time interval when the information + gets logged: + var cluster = new couchbase.Cluster("couchbase://127.0.0.1/default?tracing_threshold_queue_flush_interval=10.0"); + Follow along with our example of , to discover how to use tracing in the node.js SDK for Response Time Observability. +

      +
      + +       diff --git a/content/sdk/nodejs/tracing-from-the-sdk.dita b/content/sdk/nodejs/tracing-from-the-sdk.dita new file mode 100644 index 0000000000..5b367d8dbd --- /dev/null +++ b/content/sdk/nodejs/tracing-from-the-sdk.dita @@ -0,0 +1,57 @@ + + + + Tracing from the Node.js SDK with Couchbase Server + Tracing from the SDK + Use tracing from the SDK to discover timeouts and bottlenecks across the network and cluster. + +

      For our example, we will customize the threshold logging tracer settings with lcb_cntl. By default it + will log every 10 seconds (if something is found) and only sample the top 10 slowest operations. The default threshold for key/value operation is 500 milliseconds. We shall + set them so low that almost everything gets logged - not something you should do in production!// set kv threshold to 1 microsecond +// lower flush interval to 100 microseconds +var cluster = new couchbase.Cluster("couchbase://localhost/travel-sample?tracing_threshold_kv=0.000001&tracing_threshold_queue_flush_interval=0.0001"); + +This will set our threshold for key/value operations to one microsecond, and log the found operations every second.

      + +

      With these configs in place we are ready to run some operations. Below, we read some documents from the travel-sample bucket and, if found, write them back with an upsert - giving us both read and write operations to log.couchbase.logging.enableLogging(); + +var cluster = new couchbase.Cluster("couchbase://localhost/travel-sample?tracing_threshold_kv=0.000001&tracing_threshold_queue_flush_interval=0.0001"); +cluster.authenticateAs('Administrator', 'password'); +var bucket = cluster.openBucket('travel-sample'); + +for (var i = 0; i < 5; ++i) { + bucket.get('airline_10', function(err, res) { + if (err) { + throw err; + } + + // completed, it should be in our traces! + }); +}

      + +

      Run the code, and you will see something like the following in the logs:22ms [I3fc6ae73] {27957} [INFO] (tracer - L:148) Operations over threshold: {"count":6,"service":"kv","top":[{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x0","last_remote_address":"192.168.1.12:11210","server_us":27,"total_us":1761},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x1","last_remote_address":"192.168.1.12:11210","server_us":8,"total_us":2189},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x2","last_remote_address":"192.168.1.12:11210","server_us":3,"total_us":2356},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x3","last_remote_address":"192.168.1.12:11210","server_us":3,"total_us":2477},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"upsert:0x5","last_remote_address":"192.168.1.12:11210","server_us":100,"total_us":2489},{"last_local_address":"192.168.1.10:56762","last_local_id":"000000003fc6ae73/277953824e30913b","last_operation_id":"get:0x4","last_remote_address":"192.168.1.12:11210","server_us":3,"total_us":2592}]}For each service (only kv-based on this workload), the threshold log reporter will show you the total number of recorded ops (through count), and give you the top slowest ops sorted by their latency. Since only airline_10 exists in the bucket you will see five document fetches but only one mutation.

      + + +
      + + Output fields in detail. + +

      Let's highlight a single operation, and explain each field in a little more detail:{ + "total_us": 1761, + "server_us": 27, + "last_remote_address": "192.168.1.12:11210", + "last_operation_id": "get:0x0", + "last_local_id": "000000003fc6ae73/277953824e30913b", + "last_local_address": "192.168.1.10:56762" +}

      +

      This tells us the following:

        +
      • total_us: The total time it took to perform the full operation: here around 1.8 milliseconds.
        • +
      • server_us: The server reported that its work performed took 27 microseconds (this does not include network time or time in the buffer before picked up at the cluster).
        • +
      • last_local_address: The local socket used for this operation.
        • +
      • last_remote_address: The remote socket on the server used for this operation. Useful to figure out which node is affected.
        • +
      • last_operation_id: A combination of type of operation and id (in this case the opaque value), useful for diagnosing and troubleshooting in combination with the last_local_id.
        • +
      • last_local_id: With Server 5.5 and later, this id is negotiated with the server and can be used to correlate logging information on both sides in a simpler fashion.

      +

      You can see that if the thresholds are set the right way based on production requirements, without much effort slow operations can be logged and pinpointed more easily than before.

      +
      + +       diff --git a/content/sdk/php.ditamap b/content/sdk/php.ditamap index 0bb5105190..5a8c35b82e 100644 --- a/content/sdk/php.ditamap +++ b/content/sdk/php.ditamap @@ -2,11 +2,11 @@ PHP SDK Guide - + - 2.4.2 + 2.4.4 @@ -26,6 +26,9 @@ + + + @@ -35,7 +38,9 @@ - + + + @@ -47,6 +52,7 @@ + @@ -58,5 +64,7 @@ + + diff --git a/content/sdk/php/compatibility-versions-features.dita b/content/sdk/php/compatibility-versions-features.dita index 88649eadf7..563918e9b6 100644 --- a/content/sdk/php/compatibility-versions-features.dita +++ b/content/sdk/php/compatibility-versions-features.dita @@ -40,22 +40,6 @@ - - CB 2.5 - ◎ - ◎ - ◎ - ◎ - ◎ - - - CB 3.0 - ◎ - ◎ - ◎ - ◎ - ◎ - CB 3.1 ◎ @@ -65,40 +49,16 @@ ✔ - CB 4.0 - ✔ - ✔ - ✔ - ✔ - ✔ - - - CB 4.1 - ✔ - ✔ - ✔ - ✔ - ✔ - - - CB 4.5 - ✔ - ✔ - ✔ - ✔ - ✔ - - - CB 4.6 - ✔ + CB 4.0 - 4.6 + ◎ ✔ ✔ ✔ ✔ - CB 5.0 - ✔ + CB 5.0 - 5.1 + ◎ ✔ ✔ ✔ diff --git a/content/sdk/php/encrypting-using-sdk.dita b/content/sdk/php/encrypting-using-sdk.dita new file mode 100644 index 0000000000..06e87cbb76 --- /dev/null +++ b/content/sdk/php/encrypting-using-sdk.dita @@ -0,0 +1,105 @@ + + + + Field Level Encryption from the PHP SDK + Field Level Encryption is available in Couchbase Data Platform 5.5, from PHP SDK version 2.4.6. + +

      Having examined the concepts behind Field Level Encryption, let's see some example code to encrypt some data with the PHP SDK.

      + + +
      + Installing +

      You can download the PHP ecryption API for Couchbase from Packagist. Use composer require couchbase/couchbase-encryption or install automatically with the demo.

      + +

      The demo examples are at , install with:pecl install couchbase # make sure couchbase 2.4.6+ SDK installed +composer install +composer demo-symmetric +composer demo-asymmetric

      +
      + + +
      + Encryption in Action +

      Here is the first sample, which shows an AES-256 encryption of a single field:<?php +require __DIR__ . '/vendor/autoload.php'; +use Couchbase\Aes256HmacSha256Provider; +use Couchbase\KeyProvider; +use Couchbase\Cluster; +use Couchbase\Bucket; +final class InsecureKeyProvider implements KeyProvider +{ + public function getKey(string $id) + { + switch ($id) { + case 'mypublickey': + return "!mysecretkey#9^5usdk39d&dlf)03sL"; + case 'HMAC_KEY_ID': + return 'myauthpassword'; + default: + throw new InvalidArgumentException("Unknown key '$id"); + } + } +} +$cluster = new Cluster('couchbase://localhost'); +$cluster->authenticateAs('Administrator', 'password'); +$bucket = $cluster->openBucket('default'); +$bucket->registerCryptoProvider( + 'AES-256-HMAC-SHA256', + new Aes256HmacSha256Provider(new InsecureKeyProvider(), "mypublickey", "HMAC_KEY_ID") +); +// source document, which contains some sensitive data +$document = [ + 'message' => 'The old grey goose jumped over the wrickety gate.' +]; +// let's encrypt everything stored in the 'message' property using a key with ID +// 'mypublickey' and our crypto provider. +$encrypted = $bucket->encryptFields( + $document, + [ + [ + 'name' => 'message', + 'alg' => 'AES-256-HMAC-SHA256' + ] + ] +); +// now we are ready to persist document with encrypted field using regular APIs +$bucket->upsert('secret-1', $encrypted); +// the database does not have our encryption keys, and cannot see the plain contents +$document = $bucket->get('secret-1')->value; +var_dump($document); +// the output should be similar to following: +// => object(stdClass)#9 (1) { +// ["__crypt_message"]=> +// object(stdClass)#8 (5) { +// ["alg"]=> +// string(19) "AES-256-HMAC-SHA256" +// ["ciphertext"]=> +// string(88) "aK1RxvZkP4YWyMapQTpiRKvAG6V1MsFWUJwNfY7TXh3d5DdFO3jwmQu3rFMN6p98Y4ziM+pQNkrB/Cc7GP9/yw==" +// ["iv"]=> +// string(24) "1tzdgObtNJmNOrgSImzdKg==" +// ["kid"]=> +// string(11) "mypublickey" +// ["sig"]=> +// string(44) "qStQ7U28A05nz/ZP5SKDSMQuMofy1K9QHX8nYALLwOo=" +// } +// } +$decrypted = $bucket->decryptFields( + $document, + [ + [ + 'name' => 'message', + 'alg' => 'AES-256-HMAC-SHA256' + ] + ] +); +var_dump($decrypted); +// now we have our document readable +// => array(1) { +// ["message"]=> +// string(49) "The old grey goose jumped over the wrickety gate." +// }Try the other demo example at .

      +
      + +       +       \ No newline at end of file diff --git a/content/sdk/php/full-text-searching-with-sdk.dita b/content/sdk/php/full-text-searching-with-sdk.dita index 6bd96f830e..5130d9c43a 100644 --- a/content/sdk/php/full-text-searching-with-sdk.dita +++ b/content/sdk/php/full-text-searching-with-sdk.dita @@ -237,5 +237,261 @@ var_dump($result->facets); } }
    + + +
    Detailed Examples + +

    + The code example below demonstrates the PHP SDK Full Text Search + API. The example assumes that Couchbase Server is running, and that + the username Administrator and + the password password provide authorization for performing the searches. + It also assumes that the travel-sample bucket + has been installed. + For information on creating users and managing roles, see + Authorization. + For information on installing sample buckets, see + Settings. +

    + +

    + The example also assumes the existence of three specific Full Text Indexes, defined on the + travel-sample bucket. These are: +

    + +
      +
    • + travel-sample-index-unstored: Uses only the default + settings. +
      • + +
    • + travel-sample-index-stored: Uses default settings, with one exception: + dynamic fields are stored, for the whole index. +
      • + +
    • + travel-sample-index-hotel-description: Indexes only the description + fields of hotel + documents, and disables the default type mapping. The index + has a custom analyzer named myUnicodeAnalyzer defined on it: the analyzer's + main characteristic is that it uses the unicode tokenizer. +
      • +
    + +

    + See + Creating Indexes + for details on how to create these indexes: they can be created interactively, by + means of the Couchbase Web Console; however, there may be greater efficiency + in using the Couchbase REST + API, as described + in the section + Index Creation with the REST API. + The JSON objects that constitute index-definitions (for + inclusion as bodies to the index-creation REST calls), are provided in + Demonstration Indexes. +

    + <?php + +use \Couchbase\Cluster; +use \Couchbase\Bucket; +use \Couchbase\SearchQuery; + +function printResult($label, $resultObject) { + echo("\n"); + echo("= = = = = = = = = = = = = = = = = = = = = = =\n"); + echo("= = = = = = = = = = = = = = = = = = = = = = =\n"); + echo("\n"); + echo($label); + echo(' (total hits: ' . $resultObject->metrics['total_hits'] . ')'); + echo("\n"); + + foreach ($resultObject->hits as $row) { + echo("id=" . $row->id . ", score=" . $row->score); + if (property_exists($row, 'fields')) { + echo(", fields=" . json_encode($row->fields)); + } + if (property_exists($row, 'locations')) { + echo(", locations=" . json_encode($row->locations)); + } + if (property_exists($row, 'fragments')) { + echo(", fragments=" . json_encode($row->fragments)); + } + echo("\n"); + } +} + +// Simple Text Query on a single word, targeting an index with dynamic fields +// unstored. +function simpleTextQuery(Bucket $bucket) { + $indexName = "travel-sample-index-unstored"; + $query = new SearchQuery($indexName, + SearchQuery::match("swanky")); + $query->limit(10); + $result = $bucket->query($query); + printResult("Simple Text Query", $result); +} + +// Simple Text Query on Stored Field, specifying the field to be searched; +// targeting an index with dynamic fields stored, to ensure that field-content +// is included in the return object. +function simpleTextQueryOnStoredField(Bucket $bucket) { + $indexName = "travel-sample-index-stored"; + $query = new SearchQuery($indexName, + SearchQuery::match("MDG")->field("destinationairport")); + $query->limit(10)->highlight(SearchQuery::HIGHLIGHT_HTML); + $result = $bucket->query($query); + printResult("Simple Text Query on Stored Field", $result); +} + +// Simple Text Query on Non-Default Index, specifying an index that consists +// only of content derived from a specific field from a specific document-type. +function simpleTextQueryOnNonDefaultIndex(Bucket $bucket) { + $indexName = "travel-sample-index-hotel-description"; + $query = new SearchQuery($indexName, + SearchQuery::match("swanky")); + $query->limit(10); + $result = $bucket->query($query); + printResult("Simple Text Query on Non-Default Index", $result); +} + +// Match Query with Facet, showing how query-results can be displayed either by +// row or by hits; and demonstrating use of a facet, which provides +// aggregation-data. +function textQueryOnStoredFieldWithFacet(Bucket $bucket) { + $indexName = "travel-sample-index-stored"; + $query = new SearchQuery($indexName, + SearchQuery::match("La Rue Saint Denis!!")->field("reviews.content")); + $query->limit(10)->highlight(SearchQuery::HIGHLIGHT_HTML); + $query->addFacet('Countries Referenced', SearchQuery::termFacet('country', 5)); + $result = $bucket->query($query); + printResult("Match Query with Facet, Result by Row", $result); + + echo("Match Query with Facet, Result by facet:\n"); + echo(json_encode($result->facets) . "\n"); +} + +// DocId Query, showing results of a query on two document IDs. +function docIdQueryMethod(Bucket $bucket) { + $indexName = "travel-sample-index-unstored"; + $query = new SearchQuery($indexName, + SearchQuery::docId("hotel_26223", "hotel_28960")); + $result = $bucket->query($query); + printResult("DocId Query", $result); +} + +// Unanalyzed Term Query with Fuzziness Level, demonstrating how to query on a +// term with no analysis. Zero fuzziness is specified, to ensure that matches +// are exact. With a fuzziness factor of 2, allowing partial matches to be made. +function unAnalyzedTermQuery(Bucket $bucket, int $fuzzinessLevel) { + $indexName = "travel-sample-index-stored"; + $query = new SearchQuery($indexName, + SearchQuery::term("sushi")->field("reviews.content")->fuzziness($fuzzinessLevel)); + $query->limit(50)->highlight(SearchQuery::HIGHLIGHT_HTML); + $result = $bucket->query($query); + printResult("Unanalyzed Term Query with Fuzziness Level of " . $fuzzinessLevel . ":", $result); +} + +// Match Phrase Query, using Analysis, for searching on a phrase. +function matchPhraseQueryOnStoredField(Bucket $bucket) { + $indexName = "travel-sample-index-stored"; + $query = new SearchQuery($indexName, + SearchQuery::matchPhrase("Eiffel Tower")->field("description")); + $query->limit(10)->highlight(SearchQuery::HIGHLIGHT_HTML); + $result = $bucket->query($query); + printResult("Match Phrase Query, using Analysis", $result); +} + +// Phrase Query, without Analysis, for searching on a phrase without analysis supported. +function unAnalyzedPhraseQuery(Bucket $bucket) { + $indexName = "travel-sample-index-stored"; + $query = new SearchQuery($indexName, + SearchQuery::phrase("dorm", "rooms")->field("description")); + $query->limit(10)->highlight(SearchQuery::HIGHLIGHT_HTML); + $result = $bucket->query($query); + printResult("Phrase Query, without Analysis", $result); +} + +// Conjunction Query, whereby two separate queries are defined and then run as +// part of the search, with only the matches returned by both included in the +// result-object. +function conjunctionQuery(Bucket $bucket) { + $indexName = "travel-sample-index-stored"; + $firstQuery = SearchQuery::match("La Rue Saint Denis!!")->field("reviews.content"); + $secondQuery = SearchQuery::match("boutique")->field("description"); + $query = new SearchQuery($indexName, + SearchQuery::conjuncts($firstQuery, $secondQuery)); + $query->limit(10)->highlight(SearchQuery::HIGHLIGHT_HTML); + $result = $bucket->query($query); + printResult("Conjunction Query", $result); +} + +// Query String Query, showing how a query string is specified as search-input. +function queryStringQuery(Bucket $bucket) { + $indexName = "travel-sample-index-unstored"; + $query = new SearchQuery($indexName, + SearchQuery::queryString("description: Imperial")); + $query->limit(10); + $result = $bucket->query($query); + printResult("Query String Query", $result); +} + +// Wild Card Query, whereby a wildcard is used in the string submitted for the +// search. +function wildCardQuery(Bucket $bucket) { + $indexName = "travel-sample-index-stored"; + $query = new SearchQuery($indexName, + SearchQuery::wildcard("bouti*ue")->field("description")); + $query->limit(10)->highlight(SearchQuery::HIGHLIGHT_HTML); + $result = $bucket->query($query); + printResult("Wild Card Query", $result); +} + +// Numeric Range Query, whereby minimum and maximum numbers are specified, and +// matches within the range returned. +function numericRangeQuery(Bucket $bucket) { + $indexName = "travel-sample-index-unstored"; + $query = new SearchQuery($indexName, + SearchQuery::numericRange()->min(10100)->max(10200)->field("id")); + $query->limit(10); + $result = $bucket->query($query); + printResult("Numeric Range Query", $result); +} + +// Regexp Query, whereby a regular expression is submitted, to generate the +// conditions for successful matches. +function regexpQuery(Bucket $bucket) { + $indexName = "travel-sample-index-stored"; + $query = new SearchQuery($indexName, + SearchQuery::regexp("[a-z]")->field("description")); + $query->limit(10)->highlight(SearchQuery::HIGHLIGHT_HTML); + $result = $bucket->query($query); + printResult("Regexp Query", $result); +} + +$cluster = new Cluster('couchbase://localhost'); +$cluster->authenticateAs('Administrator', 'password'); +$bucket = $cluster->openBucket('travel-sample'); + +simpleTextQuery($bucket); +simpleTextQueryOnStoredField($bucket); +simpleTextQueryOnNonDefaultIndex($bucket); +textQueryOnStoredFieldWithFacet($bucket); +docIdQueryMethod($bucket); +unAnalyzedTermQuery($bucket, 0); +unAnalyzedTermQuery($bucket, 2); +matchPhraseQueryOnStoredField($bucket); +unAnalyzedPhraseQuery($bucket); +conjunctionQuery($bucket); +queryStringQuery($bucket); +wildCardQuery($bucket); +numericRangeQuery($bucket); +regexpQuery($bucket); + + +
    + + - + \ No newline at end of file diff --git a/content/sdk/php/managing-clusters.dita b/content/sdk/php/managing-clusters.dita index 96da28b93d..4b5a48c369 100644 --- a/content/sdk/php/managing-clusters.dita +++ b/content/sdk/php/managing-clusters.dita @@ -166,10 +166,54 @@ $designDoc = $manager->getDesignDocument('landmarks'); $designDoc['views']['by_country'] += array('reduce' => "_count"); //added reduce function // Resend to server -$bucketManager->upsertDesignDocument($designDoc); +$bucketManager->upsertDesignDocument($designName, $designDoc);

    To remove a design document from a bucket, call the removeDesignDocument method with the name of the design document.

    • + + +
    + + + Configuration Cache File + + +

    + The legacy client configuration cache file records the current cluster configuration, which + Couchbase Server uses at startup. +

    + +

    + Client applications use libcouchbase to connect to Couchbase Server. By means of libcouchbase, + the client can react to changes in cluster-topology. Whenever the client is started, libcouchbase is + instantiated, and checks the local filesystem for a configuration cache file, containing information on the + cluster-toplogy. If no such file exists, libcouchbase + connects to the cluster by means of a REST method, in order to receive the latest topology details. + Once acquired, these details are written to the configuration cache file, locally. +

    + +

    + Subsequently, when starting, the client uses the local configuration cache file to acquire its topology data. If at any time, when + using this data, the client + attempts to access a data item on a node, and is informed by Couchbase Server that the node is + incorrect (due to the cluster configuration having changed), the current configuration cache file is duly invalidated. The + client then requests an updated set of + toplogy details from Couchbase Server; which are themselves then saved locally by the client, as a + new configuration cache file. +

    + +

    + Use of the configuration cache-file is available only through + PHP (not through Java or .NET), Node.js, and Python. + Therefore, to use the cache file, add the following + to the php.ini: +

    + + couchbase.config_cache = /tmp + +
    + + diff --git a/content/sdk/php/managing-connections.dita b/content/sdk/php/managing-connections.dita index 2ec1c29f76..d2042fcd5b 100644 --- a/content/sdk/php/managing-connections.dita +++ b/content/sdk/php/managing-connections.dita @@ -45,7 +45,11 @@ $cluster->manager()->createBucket('hello'); // automatically use admin credentia which will take precedence over the authenticator. Such behaviour is discouraged, and future implementations of the \Couchbase\Authenticator interface are likely to prevent mixing of authentication sources.

    -
    + +

    You can shortcut using the authenticator() method, if you don't need to reuse the credentials in + several instances of authenticate(), replacing it with authenticateAs():$cluster = new CouchbaseCluster("couchbase://localhost"); +$cluster->authenticateAs('Administrator', 'password');

    diff --git a/content/sdk/php/n1ql-queries-with-sdk.dita b/content/sdk/php/n1ql-queries-with-sdk.dita index 4738e4c599..3147c46d5b 100644 --- a/content/sdk/php/n1ql-queries-with-sdk.dita +++ b/content/sdk/php/n1ql-queries-with-sdk.dita @@ -6,7 +6,7 @@ Perform Couchbase Query Language (N1QL) queries via the PHP SDK. See Couchbase Developer documentation for a quick intro to N1QL.

    To issue N1QL queries, you should create a CouchbaseN1qlQuery object, and pass @@ -66,6 +66,8 @@ foreach ($result->rows as $row) { prepared statement at the SDK level. If this is set to false then a prepared reference to this query will be stored on the SDK and used for subsequent queries. +

  • max_parallelism sets the number of logical CPUs which are used in parallel for the query, this setting can override the default (although only in the Enterprise Edition), but please contact before changing this on a production system. It can be used as follows:$query= CouchbaseN1qlQuery::fromString('SELECT * FROM default'); +$query->options['max_parallelism'] = "0";

    • Cross-bucket queries

    Since version 2.3.0, the PHP SDK implements the \Couchbase\Authenticator interface, diff --git a/content/sdk/php/sample-app-backend.dita b/content/sdk/php/sample-app-backend.dita index 2ba9e34bda..6dfa88e465 100644 --- a/content/sdk/php/sample-app-backend.dita +++ b/content/sdk/php/sample-app-backend.dita @@ -25,7 +25,7 @@

    The source code for the tutorial is available on GitHub, at - couchbaselabs/try-cb-php. + couchbaselabs/try-cb-php. The primary focus of the tutorial is to explain the function and theory behind the Couchbase PHP client, and how it works together with Couchbase Server; with special reference to features such as N1QL, @@ -188,7 +188,7 @@ DB_PASSWORD=travel-sample

    Note that a detailed functional description of buckets is provided in - Buckets. + Buckets.

    diff --git a/content/sdk/php/scan-consistency-examples.dita b/content/sdk/php/scan-consistency-examples.dita new file mode 100644 index 0000000000..f317bb4b4b --- /dev/null +++ b/content/sdk/php/scan-consistency-examples.dita @@ -0,0 +1,47 @@ + + + + Scan Consistency Using the PHP SDK with Couchbase Server + Using Scan Consistency + You can balance performance against consistency in N1QL queries via the Couchbase PHP client and the AtPlus option. + +

    + Setting a Scan Consistency, lets you choose between NOT_BOUNDED (the default), for sharpest performance; REQUEST_PLUS, for strongest consistency; and AT_PLUS, for a good balance between increased performance and completeness of results. +

    + +

    The following example — for which mutation tokens must be enabled during connection — shows AT_PLUS in action:<?php + +$cluster = new \Couchbase\Cluster('couchbase://192.168.1.194?fetch_mutation_tokens=true'); +$bucket = $cluster->openBucket('default'); +$RANDOM_NUMBER = rand(0, 10000000); +$result = $bucket->upsert('user:'.$RANDOM_NUMBER, array( + "name" => array("Brass", "Doorknob"), + "email" => "brass.doorknob@example.com", + "random" => $RANDOM_NUMBER) +); +// construct mutation state from the list of mutation results +$mutationState = \Couchbase\MutationState::from($result); +$query = \Couchbase\N1qlQuery::fromString( + 'SELECT name, email, random, META(default).id FROM default WHERE $1 IN name' +); +$query->positionalParams(['Brass']); +// If this line is removed, the latest 'random' field might not be present +$query->consistentWith($mutationState); +printf("Expecting random: %d\n", $RANDOM_NUMBER); +$result = $bucket->query($query); +foreach ($result->rows as $row) { + printf("Name: %s, Email: %s, Random: %d\n", implode(" ", $row->name), $row->email, $row->random); + if ($row->random == $RANDOM_NUMBER) { + echo "!!! Found or newly inserted document !!!\n"; + } + if (getenv("REMOVE_DOORKNOBS")) { + echo "Removing " . $row->id . " (Requested via env)\n"; + $bucket->remove($row->id); + } +}

    +
    + +     +     \ No newline at end of file diff --git a/content/sdk/php/sdk-authentication-overview.dita b/content/sdk/php/sdk-authentication-overview.dita index 8c3e257327..d85e3eb148 100644 --- a/content/sdk/php/sdk-authentication-overview.dita +++ b/content/sdk/php/sdk-authentication-overview.dita @@ -101,6 +101,40 @@
    +
    + + Certificate-Based Authentication + + +

    + Couchbase Server supports the use of X.509 certificates to authenticate clients (only available in + the Enterprise Edition, not the Community Edition). This allows + authenticated users to access specific resources by means of the data service (no other service + is currently available to clients through this form of authentication). +

    + +

    + The process relies on a certificate + authority, for the issuing of certificates that validate identities. A certificate includes + information such as the name of the entity it identifies, an expiration date, the name of the authority + that issued the certificate, and the digital signature of the authority. + A client attempting to access Couchbase Server can present a certificate to the server, allowing + the server to check the validity of the certificate. If the certificate is valid, the user + under whose identity the client is running, + and the roles assigned that user, are verified. If the assigned roles are appropriate for + the level of access requested to the specified resource, access is granted. +

    + +

    The PHP SDK has supported certificate authentication since version 2.4.3. The client certificate can be generated with the shell script on + the C SDK Authentication page. Note + that you will need to load the php_openssl module before php_couchbase in your code.

    +

    + You can now connect with the following in your <php + block:$cluster = new \Couchbase\Cluster('couchbases://localhost' . '?keypath=/tmp/foo.key&certpath=/tmp/foo.cert'); + $cluster->authenticate(new \Couchbase\CertAuthenticator()); + $bucket = $cluster->openBucket('default');

    +
    +     diff --git a/content/sdk/php/sdk-user-management-overview.dita b/content/sdk/php/sdk-user-management-overview.dita index a4ef325681..8bd793fab8 100644 --- a/content/sdk/php/sdk-user-management-overview.dita +++ b/content/sdk/php/sdk-user-management-overview.dita @@ -66,8 +66,7 @@
  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    diff --git a/content/sdk/php/start-using-sdk.dita b/content/sdk/php/start-using-sdk.dita index 2f20473d87..db081450bd 100644 --- a/content/sdk/php/start-using-sdk.dita +++ b/content/sdk/php/start-using-sdk.dita @@ -36,6 +36,11 @@ sudo apt-get update sudo apt-get install libcouchbase-dev build-essential php-dev zlib1g-dev sudo pecl install couchbase + On Ubuntu 14.04 (Trusty), and possibly some other distributions, you will need to substitute + for php-dev with php5-dev. To find particular variants of php-dev for your Linux distribution, + use the command $ apt-cache search php | grep dev + + # Only needed during first-time setup: wget http://packages.couchbase.com/releases/couchbase-release/couchbase-release-1.0-4-x86_64.rpm sudo rpm -iv couchbase-release-1.0-4-x86_64.rpm @@ -46,15 +51,9 @@ sudo pecl install couchbase

    • -
    Installation on Mac OS X

    To install the library on Mac OS X, first install the de-facto package manager for OS X: homebrew. Once - homebrew is - configured:brew update # get list of latest packages -brew install libcouchbase -# brew install homebrew/php/php{XY}-couchbase, where XY is your version of PHP -brew install homebrew/php/php70-couchbase # for PHP 7.0

    If - you have PECL installed, you may use pecl install couchbase to install - the Couchbase PHP SDK.
    +
    Installation on Mac OS X +

    As homebrew/php is no longer available, the current best way to install newer versions of PHP (and one with the necessary permissions) is found at . Download install.sh from there or, if you are happy with the script, install in one line (in this example, PHP 7.2) with:curl -s https://php-osx.liip.ch/install.sh | bash -s 7.2

    +

    Once you have PHP installed, then use PECL to install the Couchbase PHP SDK:pecl install couchbase

    Installing on Microsoft Windows

    When using Microsoft Windows, download one of the pre-built binaries at Archives, @@ -156,7 +155,7 @@ echo "Creating primary index\n"; // Before issuing a N1QL Query, ensure that there is // is actually a primary index. try { - // Do not override default name, fail if it is exists already, and wait for completion + // Do not override default name; fail if it already exists, and wait for completion $bucket->manager()->createN1qlPrimaryIndex('', false, false); echo "Primary index has been created\n"; } catch (CouchbaseException $e) { diff --git a/content/sdk/php/threshold-logging.dita b/content/sdk/php/threshold-logging.dita new file mode 100644 index 0000000000..6d319aa28f --- /dev/null +++ b/content/sdk/php/threshold-logging.dita @@ -0,0 +1,122 @@ + + + + Threshold Logging Tracing through the SDK + Threshold Logging + + + +

    +
    + +
    +
    + +
    + + Threshold Logging in PHP SDK + + +

    + Response Time Observability is implemented as Threshold Logging in the Couchbase PHP SDK from release 2.5.0.

    + + + Threshold Logging Tracer Properties + + + + Setting name + Description + Default Value + + + + + enable_tracing + Boolean used to determine tracing is enabled. Defaults to using the + ThesholdLoggingTracer if enabled. When false a Noop or similar tracing + implementation should be used instead. Also when enabled the SDK will try and retrieve + duration metrics from the server for KV operations. + true + + + tracing_threshold_queue_flush_interval + The interval between executions that process the collected operation spans. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 10.0 seconds (or 10000000 microseconds) + + + tracing_threshold_queue_size + The maximum number of items to keep in internal queue per service. + 128 + + + tracing_threshold_kv + The KV operation operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 0.5 second (500000 microseconds) + + + tracing_threshold_view + The View query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_n1ql + The N1QL query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_fts + The FTS query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_threshold_analytics + The Analytics query operation threshold. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 1 second (1000000 microseconds) + + + tracing_orphaned_queue_flush_interval + The interval between executions that processes the collected operation spans. + Expressed in seconds with fractions (or microseconds for lcb_cntl). + + 10.0 seconds (or 10000000 microseconds) + + + tracing_orphaned_queue_size + The maximum number of items to keep in internal queue per service. + 128 + + + +
    +
    + + +
    + + Customizing + +

    + You can customise Tracer settings through the connection string, or using lcb_cntl. Here + is an example on how to customize our default tracer to reduce the time interval when the information + gets logged: + +$cluster = new \Couchbase\Cluster("couchbase://127.0.0.1/default?tracing_threshold_queue_flush_interval=10.0"); + + Follow along with our example of , to discover how to use tracing in the PHP SDK for Response Time Observability. +

    +
    + + diff --git a/content/sdk/php/tracing-from-the-sdk.dita b/content/sdk/php/tracing-from-the-sdk.dita new file mode 100644 index 0000000000..4599a59de2 --- /dev/null +++ b/content/sdk/php/tracing-from-the-sdk.dita @@ -0,0 +1,64 @@ + + + + Tracing from the PHP SDK with Couchbase Server + Tracing from the SDK + Use tracing from the SDK to discover timeouts and bottlenecks across the network and cluster. + +

    For our example, we will customize the threshold logging tracer settings with connection string. By default it + will log every 10 seconds (if something is found) and only sample the top 10 slowest operations. The default threshold for key/value operation is 500 milliseconds. We shall + set them so low that almost everything gets logged - not something you should do in production! +// set kv threshold to 1 microsecond +// lower flush interval to 100 microseconds +$cluster = new \Couchbase\Cluster("couchbase://localhost/travel-sample?tracing_threshold_kv=0.000001&tracing_threshold_queue_flush_interval=0.0001"); +This will set our threshold for key/value operations to one microsecond, and log the found operations every second.

    + +

    With these configs in place we are ready to run some operations. Below, we read some documents from the travel-sample bucket and, if found, write them back with an upsert - giving us both read and write operations to log.<?php + +ini_set("couchbase.log_level", "INFO"); +ini_set("error_log", ""); // will use default logging device +// set kv threshold to 1 microsecond + + +$cluster = new \Couchbase\Cluster("couchbase://localhost/travel-sample?tracing_threshold_kv=0.000001&tracing_threshold_queue_flush_interval=0.0001"); +$cluster->authenticateAs('Administrator', 'password'); +$bucket = $cluster->openBucket('travel-sample'); + +for ($i = 0; $i < 5; $i++) { + try { + $id = "airline_1" . $i; + $res = $bucket->get($id); + $bucket->upsert($id, $res->value); + } catch (Couchbase\Exception $ex) { + if ($ex->getCode() != COUCHBASE_KEY_ENOENT) { + throw new Exception("Unexpected Couchbase exception", $ex); + } + } +}

    + +

    Run the code, and you will see something like the following in the logs:[cb,INFO] (tracer L:151 I:2265433480) Operations over threshold: {"count":3,"service":"kv","top":[{"last_local_address":"127.0.0.1:38374","last_local_id":"000000008707c588/71cf014552d1ebf7","last_operation_id":"get:0x2","last_remote_address":"localhost:11210","server_us":11,"total_us":155},{"last_local_address":"127.0.0.1:38374","last_local_id":"000000008707c588/71cf014552d1ebf7","last_operation_id":"get:0x0","last_remote_address":"localhost:11210","server_us":23,"total_us":230},{"last_local_address":"127.0.0.1:38374","last_local_id":"000000008707c588/71cf014552d1ebf7","last_operation_id":"upsert:0x1","last_remote_address":"localhost:11210","server_us":76,"total_us":253}]}For each service (only kv-based on this workload), the threshold log reporter will show you the total number of recorded ops (through count), and give you the top slowest ops sorted by their latency. Since only airline_10 exists in the bucket you will see up to five document fetches but only one mutation. In our case we've got 2 gets and 1 upsert (depends on flush interval).

    + + +
    + + Output fields in detail. + +

    Let's highlight a single operation, and explain each field in a little more detail:{ + "last_local_address": "127.0.0.1:38374", + "last_local_id": "000000008707c588/71cf014552d1ebf7", + "last_operation_id": "get:0x2", + "last_remote_address": "localhost:11210", + "server_us": 11, + "total_us": 155 +}

    +

    This tells us the following:

      +
    • total_us: The total time it took to perform the full operation: here around 0.1 milliseconds (so fast because server is deployed on localhost).
      • +
    • server_us: The server reported that its work performed took 11 microseconds (this does not include network time or time in the buffer before picked up at the cluster).
      • +
    • last_local_address: The local socket used for this operation.
      • +
    • last_remote_address: The remote socket on the server used for this operation. Useful to figure out which node is affected.
      • +
    • last_operation_id: A combination of type of operation and id (in this case the opaque value), useful for diagnosing and troubleshooting in combination with the last_local_id.
      • +
    • last_local_id: With Server 5.5 and later, this id is negotiated with the server and can be used to correlate logging information on both sides in a simpler fashion.

    +

    You can see that if the thresholds are set the right way based on production requirements, without much effort slow operations can be logged and pinpointed more easily than before.

    +
    + +     diff --git a/content/sdk/python.ditamap b/content/sdk/python.ditamap index e3a2098337..8a192908f8 100644 --- a/content/sdk/python.ditamap +++ b/content/sdk/python.ditamap @@ -17,6 +17,9 @@ + + + @@ -36,6 +39,7 @@ + @@ -47,5 +51,7 @@ + + diff --git a/content/sdk/python/compatibility-versions-features.dita b/content/sdk/python/compatibility-versions-features.dita index 2264dc6a13..3ec09d62ac 100644 --- a/content/sdk/python/compatibility-versions-features.dita +++ b/content/sdk/python/compatibility-versions-features.dita @@ -38,61 +38,25 @@ - CB 2.5 - ◎ - ◎ + CB 3.1 ◎ ◎ - - - CB 3.0 ◎ - ◎ ◎ - ◎ - - - CB 3.1 - ◎ - ✔ - ✔ - ✔ - - - CB 4.0 - ✔ - ✔ - ✔ - ✔ - CB 4.1 + CB 4.0 - 4.6 ✔ ✔ ✔ ✔ - CB 4.5 - ✔ - ✔ - ✔ - ✔ - - - CB 4.6 - ✔ - ✔ - ✔ - ✔ - - - CB 5.0 - ✔ + CB 5.0 - 5.1 + ◎ ✔ ✔ ✔ - diff --git a/content/sdk/python/encrypting-using-sdk.dita b/content/sdk/python/encrypting-using-sdk.dita new file mode 100644 index 0000000000..ca23ea58dd --- /dev/null +++ b/content/sdk/python/encrypting-using-sdk.dita @@ -0,0 +1,58 @@ + + + + Field Level Encryption from the Python SDK + Field Level Encryption is available in Couchbase Data Platform 5.5, from Python SDK version 2.4.0. + +

    Having examined the concepts behind Field Level Encryption, let's follow through how we encrypt some data with the Python SDK.

    + + +
    + Keystore +

    The following example starts with importing the InMemoryKeyStore module, and creating the insecure keystore (only use this one in testing!). Then registering the provider:from cbencryption import AES256CryptoProvider +from couchbase.bucket import Bucket +from couchbase.crypto import InMemoryKeyStore +# create insecure key store and register both public and private keys +keystore = InMemoryKeyStore() +keystore.set_key('mypublickey', b'!mysecretkey#9^5usdk39d&dlf)03sL') +keystore.set_key('myprivatekey', b'myauthpassword') + +# create and register provider +provider = AES256CryptoProvider.AES256CryptoProvider(keystore, 'mypublickey', 'myprivatekey') +bucket = Bucket("couchbase://10.143.180.101:8091/default",password='password') +bucket.register_crypto_provider('AES-256-HMAC-SHA256', provider)

    + + +
    + Encryption +

    To encrypt a document, the alg name must match the provider name, and the kid (key ID) must match a key in the keystore:prefix = '__crypt_' +document = {'message': 'The old grey goose jumped over the wrickety gate.'} +fieldspec = [{'alg': 'AES-256-HMAC-SHA256', 'name': 'message'}] +encrypted_document = bucket.encrypt_fields(document, + fieldspec, + prefix) +expected = { + "__crypt_message": {"alg": "AES-256-HMAC-SHA256", + "kid": "mypublickey", + "ciphertext": "sR6AFEIGWS5Fy9QObNOhbCgfg3vXH4NHVRK1qkhKLQqjkByg2n69lot89qFEJuBsVNTXR77PZR6RjN4h4M9evg==" + } +}

    + + +
    + Decryption & Checking +

    We can filter the signature/iv-independent fields, for comparison, to check that the original and the decrypted values are the same:def filter_encrypted(encrypted_dict): + return {k:v for k,v in encrypted_dict.items() if k in {"alg","kid","ciphertext"}} + +subset_expected = filter_encrypted(expected) +subset_actual = filter_encrypted(encrypted_document) +assert subset_expected == subset_actual +# decrypt document using registered provider +decrypted_document = bucket.decrypt_fields(encrypted_document, fieldspec, prefix) +assert decrypted_document==documentThe complete + code sample is available in our devguide examples.

    +
    + +     +     \ No newline at end of file diff --git a/content/sdk/python/sample-app-backend.dita b/content/sdk/python/sample-app-backend.dita index fce70fed46..99d5b2cb5b 100644 --- a/content/sdk/python/sample-app-backend.dita +++ b/content/sdk/python/sample-app-backend.dita @@ -8,7 +8,7 @@

    The full source code for the tutorial is available on GitHub: http://github.com/couchbaselabs/try-cb-python. The primary focus of the tutorial is to explain the function and theory behind the Couchbase Python client and how it works together with Couchbase Server, and especially new features in versions 4.0/4.5 like N1QL, FTS and sub-document. It makes use of the travel-sample data set. The code that generates the web application is provided with the source code, @@ -42,13 +42,8 @@

    This tutorial focuses on querying through N1QL and FTS rather than - views. If you want information about using views, see the following resources:
      -
    • General Information about views: -
      • -
    • Querying views:
      • -
    + views. If you want information about using views, see + Views.
    Walking Through the API diff --git a/content/sdk/python/sdk-authentication-overview.dita b/content/sdk/python/sdk-authentication-overview.dita index 64c12f2ad9..85fa86c3a1 100644 --- a/content/sdk/python/sdk-authentication-overview.dita +++ b/content/sdk/python/sdk-authentication-overview.dita @@ -84,8 +84,7 @@
  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + RBAC for Applications.

    @@ -99,7 +98,63 @@

    • - +
    + + Certificate-Based Authentication + + +

    + Couchbase Server supports the use of X.509 certificates to authenticate clients (only available in + the Enterprise Edition, not the Community Edition). This allows + authenticated users to access specific resources by means of the data service (no other service + is currently available to clients through this form of authentication). +

    + +

    + The process relies on a certificate + authority, for the issuing of certificates that validate identities. A certificate includes + information such as the name of the entity it identifies, an expiration date, the name of the authority + that issued the certificate, and the digital signature of the authority. + A client attempting to access Couchbase Server can present a certificate to the server, allowing + the server to check the validity of the certificate. If the certificate is valid, the user + under whose identity the client is running, + and the roles assigned that user, are verified. If the assigned roles are appropriate for + the level of access requested to the specified resource, access is granted. +

    + +

    The Python SDK has supported certificate authentication since version 2.3.3. The client certificate can be generated with the shell script on + the C SDK Authentication page. + The following is an example of connecting with the Python client, using a certificate: + from couchbase.cluster import * +import os.path + +hostname ="localhost" +bucket_name = "default" + +# point to certificates, keys, and trust stores; +# for the purposes of this code, +# the script in etc/x509-cert will generate these + +clientdir = "etc/x509-cert/SSLCA/clientdir" +options = dict(certpath=os.path.join(clientdir, "client.pem"), + truststorepath=os.path.join(clientdir, "trust.pem"), + keypath=os.path.join(clientdir, "client.key")) + +# create a Cluster object + +cb_cluster = Cluster("http://{}/".format(hostname)) + +# create an SSL-based Authenticator +authenticator = CertAuthenticator(cluster_username="admin", + cluster_password="password", **options) +# apply this to the cluster +cb_cluster.authenticate(authenticator) + +# user this to open a bucket +cb = cb_cluster.open_bucket(bucket_name)Further examples can be seen in the Devguide examples.

    + +
    + diff --git a/content/sdk/python/sdk-user-management-overview.dita b/content/sdk/python/sdk-user-management-overview.dita index c1b16f02d5..20d2ccb4ee 100644 --- a/content/sdk/python/sdk-user-management-overview.dita +++ b/content/sdk/python/sdk-user-management-overview.dita @@ -66,8 +66,7 @@
  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    diff --git a/content/sdk/python/start-using-sdk.dita b/content/sdk/python/start-using-sdk.dita index 2a426bd92f..44c0b31d40 100644 --- a/content/sdk/python/start-using-sdk.dita +++ b/content/sdk/python/start-using-sdk.dita @@ -8,13 +8,13 @@ twisted, gevent, and asyncio. It depends on the C SDK and utilizes it for performance and reliability. -

    Installing on LinuxFor installation on Linux, install the +
    Installing on Linux

    For installation on Linux, install the couchbase-release repository, and then install the libcouchbase packages. The following examples download and install couchbase-release repsitory, a C and C++ compiler, the C SDK development files (libcouchbase-devel [RPM] or libcouchbase-dev [DEB]), Python development files, and finally the Python SDK using - pip.# Only needed during first-time setup: + pip.

    # Only needed during first-time setup: wget http://packages.couchbase.com/releases/couchbase-release/couchbase-release-1.0-4-amd64.deb sudo dpkg -i couchbase-release-1.0-4-amd64.deb sudo apt-get update @@ -34,20 +34,27 @@ sudo pip install couchbaseRHEL/CentOS

    To install the library on Mac OS X, first install the de-facto package manager for OS X: homebrew. Once homebrew is - configured:brew update # get list of latest packages + configured:

    brew update # get list of latest packages brew install libcouchbase brew install python pip install couchbaseThe above example uses the Python supplied by homebrew and not the vendor-supplied Python which ships with OS X. The Python SDK will still work with the vendor-supplied Python (though pip install may be a privileged command), - but it is recommended to use Homebrew's Python instead.

    + but it is recommended to use Homebrew's Python instead.
    -
    Installing on Microsoft WindowsDownload the executable installer - relevant to your platform and Python version from -

    Note that installation by means of pip will not work on Windows.

    +
    + Installing on Microsoft Windows +

    The Couchbase Python SDK is available as an executable installer for Windows. + This contains all the required dependencies, including C SDK development files.

    +
      +
    1. Check your Windows architecture (32-bit or 64-bit) and make sure the required version of Python is installed.
      2. +
    2. Download the Couchbase Python SDK installer for your platform and Python version from .
      3. +
    3. Run the installer and follow any instructions given.
      4. +
    + Installation by means of pip will not work on Windows.
    @@ -71,6 +78,7 @@ cb.upsert('u:king_arthur', {'name': 'Arthur', 'email': 'kingarthur@couchbase.com cb.get('u:king_arthur').value # {u'interests': [u'Holy Grail', u'African Swallows'], u'name': u'Arthur', u'email': u'kingarthur@couchbase.com'} +## The CREATE PRIMARY INDEX step is only needed the first time you run this script cb.n1ql_query('CREATE PRIMARY INDEX ON bucket-name').execute() from couchbase.n1ql import N1QLQuery row_iter = cb.n1ql_query(N1QLQuery('SELECT name FROM bucket-name WHERE ' +\ @@ -123,10 +131,9 @@ bucket = cluster.open_bucket('bucket-name')</codeblock> simple methods on the <apiname>Bucket</apiname> class such as <apiname>Bucket.get</apiname> and <apiname>Bucket.upsert</apiname>. Simply pass the key (and value, if applicable) to the relevant - methods.<codeblock outputclass="language-python">rv = bucket.get('document-id') + methods.</p><codeblock outputclass="language-python">rv = bucket.get('document-id') print(rv.value)</codeblock><codeblock outputclass="language-python">bucket.upsert('document-id', {'application': 'data'}) </codeblock> - </p> </section> @@ -138,32 +145,32 @@ bucket = cluster.open_bucket('bucket-name')</codeblock> <p> Couchbase N1QL queries are performed by creating a <apiname>N1QLQuery</apiname> object and passing that to the <apiname>Bucket.n1ql_query()</apiname> - method<codeblock outputclass="language-python">from couchbase.n1ql import N1QLQuery + method:</p><codeblock outputclass="language-python">from couchbase.n1ql import N1QLQuery query = N1QLQuery("SELECT airportname, city, country FROM `travel-sample ` "WHERE type=\"airport\" AND city=$my_city", my_city="Reno") for row in bucket.n1ql_query(query): print(row)</codeblock> - </p> </section> <section> <title> API Reference - The API reference is generated for each release and can - be found at http://pythonhosted.org/couchbase. Most of the API - documentation can also be accessed via pydoc. +

    The API reference is generated for each release and can + be found linked from the release notes for your version + of the Python SDK. Most of the API + documentation can also be accessed via pydoc.

    Release Notes - Information on new features, fixes, known issues as well +

    Information on new features, fixes, known issues as well as information on how to install older release versions is in the release notes.

    + scope="peer">in the release notes    .

    @@ -174,10 +181,10 @@ for row in bucket.n1ql_query(query): using the CPython API, the official SDK will not work on PyPy. </p> - An unofficial + <p>An unofficial module, <xref href="https://github.com/couchbaselabs/couchbase-python-cffi" format="html" scope="external">couchbase_ffi</xref> uses ffi rather than the CPython - C API to implement the internals of the library, and may be used with pypy. + C API to implement the internals of the library, and may be used with pypy.</p> </section> <section id="python-contributing"> diff --git a/content/sdk/python/threshold-logging.dita b/content/sdk/python/threshold-logging.dita new file mode 100644 index 0000000000..2c858141a1 --- /dev/null +++ b/content/sdk/python/threshold-logging.dita @@ -0,0 +1,139 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE topic PUBLIC "-//OASIS//DTD DITA Topic//EN" "topic.dtd"> +<topic id="threshold-logging"> + <title>Threshold Logging Tracing through the SDK + Threshold Logging + + + +
    +
    + +
    +
    + +
    + Threshold Configuration Settings +

    The threshold tracer receives completed spans and verifies if an operation has exceeded the + given threshold for the operation type. Operations that exceed the threshold are periodically + logged with a total count and a sample of the slowest ones. The following are Python SDK configuration properties.

    + + Threshold Logging Tracer Properties + + + + + + + Property Name + Description + Default Value + + + + + threshold_logging_tracer_interval + The interval between executions that process the collected operation spans. + Expressed in milliseconds. + + 10,000 (10 seconds) + + + + tracing_threshold_queue_size + The maximum number of items to log per service. + 10 + + + tracing_threshold_kv + The KV operation operation threshold. Expressed in microseconds. + 500,000 (500 milliseconds) + + + tracing_threshold_view + The View query operation threshold. Expressed in microseconds. + 1,000,000 (1 second) + + + tracing_threshold_n1ql + The N1QL query operation threshold. Expressed in microseconds. + 1,000,000 (1 second) + + + tracing_threshold_fts + The FTS query operation threshold. Expressed in microseconds. + 1,000,000 (1 second) + + + tracing_threshold_analytics + The Analytics query operation threshold. Expressed in microseconds. + 1,000,000 (1 second) + + + tracing_orphaned_queue_flush_interval + The interval used to flush orphaned response information to the log. Expressed in microseconds. + 10,000 (10 seconds) + + + tracing_orphaned_queue_size + The number of sample orphaned responses whose to log additional information for per execution. + 10 + + + +
    + + +
    + + +
    + + Threshold Logging in Python + + +

    + Threshold tracing is enabled by default. It can be disabled explicitly by adding enable_tracing=false + to the connection string/open_bucket options. + Here is the code to override the default values of the tracer:options = dict(hostname="localhost",bucket="default") +bucket = Bucket("couchbase://{hostname}/{bucket}".format(**options)) +bucket.tracing_threshold_queue_flush_interval = 5000 # 5 seconds +bucket.tracing_threshold_queue_size = 5 +bucket.tracing_threshold_kv = 500000 # 500 ms +bucket.tracing_threshold_n1ql = 1000000 # 1 second +bucket.tracing_threshold_view = 1000000 # 1 second +bucket.tracing_threshold_fts = 1000000 # 1 second +bucket.tracing_threshold_analytics = 1000000 # 1 second

    + +

    +In order to see warnings from the threshold tracer, you will need to enable logging:import couchbase +import logging +couchbase.enable_logging() # allows us to see warnings for slow/orphaned operations + +# set logging to WARNING level so we can see warnings +ch = logging.StreamHandler() +ch.setLevel(logging.WARNING) +logging.getLogger().addHandler(ch)

    +

    + One can also pass in an OpenTracing tracer in the Bucket options, e.g.:from opentracing_pyzipkin.tracer import Tracer +import requests + +def http_transport(encoded_span): + # The collector expects a thrift-encoded list of spans. + requests.post('http://localhost:9411/api/v1/spans', + data=encoded_span, + headers={'Content-Type': 'application/x-thrift'}) + +def jaeger_tracer(service, port = 9414, **kwargs ): + port = 9411 + tracer= Tracer("My Tracer Name", 100, http_transport, port) + return tracer + +options = dict(hostname="localhost",bucket="default") +bucket = Bucket("couchbase://{hostname}/{bucket}".format(**options), tracer = jaeger_tracer())This will also log spans to the provided tracer. At present the interface does not populate + the 'parent' field in the start_span function, so hierarchical information is lost. This + feature will be added in an upcoming Python SDK release. +

    +
    + + diff --git a/content/sdk/python/tracing-from-the-sdk.dita b/content/sdk/python/tracing-from-the-sdk.dita new file mode 100644 index 0000000000..a380f3f9d4 --- /dev/null +++ b/content/sdk/python/tracing-from-the-sdk.dita @@ -0,0 +1,292 @@ + + + + Tracing from the Python SDK with Couchbase Server + Tracing from the SDK + Use tracing from the SDK to discover timeouts and bottlenecks across the network and cluster. + +

    For our example, we will customize the threshold logging tracer settings with couchbase.Bucket. By default it + will log every minute (if something is found) and only sample the top 10 slowest operations. The default threshold for key/value operation is 500 milliseconds. We shall + set them so low that almost everything gets logged - not something you should do in production! +options = dict(bucket="default", enable_tracing="yes") +hostname ="localhost" +cluster = Cluster("couchbase://{hostname}".format(hostname=hostname)); +cluster.authenticate("username", "password"); +bucket = cluster.open_bucket(**options) +queue_size=100000000 +bucket.tracing_threshold_kv=1 +bucket.threshold_logging_tracer_interval=1e9 +bucket.tracing_threshold_queue_size=queue_size +bucket.tracing_orphaned_queue_size=queue_size +This will set our threshold for key/value operations to one microsecond, log the found operations every second, and set the sample size to a very large value so everything will be logged.

    + +

    With these configs in place we are ready to run some operations. Below, we read some documentss from the travel-sample bucket and, if found, write them back with an upsert - giving us both read and write operations to log.// Connect + + +// Load a couple of docs and write them back +for i in range(0,5): + keyname = "airline_1" + str(i) + doc = bucket.get(keyname); + if doc and doc.value: + bucket.upsert(keyname, doc.value); + +time.sleep(60)

    + +

    Run the code, and you will see something like the following in the logs:Apr 04, 2018 9:42:57 AM com.couchbase.client.core.tracing.ThresholdLogReporter logOverThreshold +[4141233127] Operations over threshold: +{ + "count": 99, + "service": "kv", + "top": [ + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1ca4", + "operation_name": "upsert", + "server_us": 19, + "total_us": 8898 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1ca7", + "operation_name": "upsert", + "server_us": 8, + "total_us": 10334 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1caa", + "operation_name": "upsert", + "server_us": 6, + "total_us": 11273 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cb0", + "operation_name": "upsert", + "server_us": 6, + "total_us": 11746 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cb4", + "operation_name": "upsert", + "server_us": 3, + "total_us": 12232 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cb6", + "operation_name": "upsert", + "server_us": 6, + "total_us": 12750 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cb9", + "operation_name": "upsert", + "server_us": 3, + "total_us": 13241 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cbd", + "operation_name": "upsert", + "server_us": 3, + "total_us": 13687 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cbf", + "operation_name": "upsert", + "server_us": 3, + "total_us": 14292 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cc4", + "operation_name": "upsert", + "server_us": 3, + "total_us": 14764 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cc8", + "operation_name": "upsert", + "server_us": 3, + "total_us": 15346 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cc9", + "operation_name": "upsert", + "server_us": 3, + "total_us": 16046 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1ccd", + "operation_name": "upsert", + "server_us": 3, + "total_us": 16602 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cd7", + "operation_name": "upsert", + "server_us": 3, + "total_us": 17054 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cdb", + "operation_name": "upsert", + "server_us": 6, + "total_us": 17622 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cdf", + "operation_name": "upsert", + "server_us": 3, + "total_us": 18123 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1ce8", + "operation_name": "upsert", + "server_us": 3, + "total_us": 18531 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cef", + "operation_name": "upsert", + "server_us": 3, + "total_us": 18987 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cf4", + "operation_name": "upsert", + "server_us": 6, + "total_us": 19523 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cfc", + "operation_name": "upsert", + "server_us": 3, + "total_us": 19894 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cfe", + "operation_name": "upsert", + "server_us": 3, + "total_us": 20497 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1cff", + "operation_name": "upsert", + "server_us": 3, + "total_us": 21137 + }, + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1d01", + "operation_name": "upsert", + "server_us": 2, + "total_us": 21838 + }, + ..., + "total_us":69233 +} + + For each service (only kv-based on this workload), the threshold log reporter will show you the total number of recorded ops (through count), and give you the top slowest ops sorted by their latency. Since only airline_10 exists in the bucket you will see five document fetches but only one mutation.

    + + +
    + + Output fields in detail. + +

    Let's highlight a single operation, and explain each field in a little more detail:{ + { + "last_local_address": "192.168.99.1:49414", + "last_local_id": "00000000f6d633e7/1816c785a1ee1ffe", + "last_remote_address": "192.168.99.1:49398", + "operation_id": "0x1d01", + "operation_name": "upsert", + "server_us": 2, + "total_us": 21838 + } +}

    +

    This tells us the following:

      +
    • total_us: The total time it took to perform the full operation: here around 21.8 milliseconds.
      • +
    • server_us: The server reported that its work performed took 2 microseconds (this does not include network time or time in the buffer before picked up at the cluster).
      • +
    • last_local_address: The local socket used for this operation.
      • +
    • last_remote_address: The remote socket on the server used for this operation. Useful to figure out which node is affected.
      • +
    • operation_id: The id (in this case the opaque value), useful for diagnosing and troubleshooting in combination with the last_local_id.
      • +
    • operation_name: The operation name, useful for diagnosing and troubleshooting in combination with the last_local_id.
      • +
    • last_local_id: With Server 5.5 and later, this id is negotiated with the server and can be used to correlate logging information on both sides in a simpler fashion.

    +

    You can see that if the thresholds are set the right way based on production requirements, without much effort slow operations can be logged and pinpointed more easily than before.

    +
    + + +
    + + Timeout Visibility. + +

    Previously, when an operation takes longer than the timeout specified allows, a TimeoutError is thrown. It usually looks like this:.root: ERROR: Got exception [<Key='key', RC=0x17[Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout], Operational Error, Results=1, C Source=(src/multiresult.c,316), Tracing Output={"key": {"s": "kv:get", "c": "00000000d5ba1b67/360fbfab17ec265c", "i": 18232203239736973189, "b": "default", "l": "10.143.180.1:57847", "r": "10.143.180.1:57834", "t": 900}}>]Now the timeout itself provides you valuable information like the local and remote sockets, and the operation id, as well as the timeout set and the local ID used for troubleshooting. You can take this information and correlate it to the top slow operations in the threshold log.

    + +

    The TimeoutError now provides you more information into what went wrong and then you can go look at the log to figure out why it was slow. +

    +
    + +     diff --git a/content/sdk/sample-application.dita b/content/sdk/sample-application.dita index 27cd148226..9647e06867 100644 --- a/content/sdk/sample-application.dita +++ b/content/sdk/sample-application.dita @@ -2,26 +2,17 @@ Sample Application - Couchbase Travel is a sample web application that demonstrates how to interact with the - Couchbase query services via the SDKs + The Couchbase Travel App is a sample web application that demonstrates how to interact with the + Couchbase query services via the SDKs. -

    Each SDK comes with its own implementation of the backend for the application. You can download - the complete source code and then build and run the app. While the app runs, you can get a peek of what is happening in - the backend via the "Narration" (a console like UI element that can display N1QL queries for instance). It is only - visible in components that communicate with the server (but can also be collapsed). -

    - -

    The documentation for the travel app illustrates the data model and walks through the N1QL and FTS queries used to - select flights and search for hotels.

    -

    The travel app front-end is the same in each SDK implementation and is built on an Angular2 stack.

    - -

    The focus of this tutorial is the backend, which is a REST API available in the following languages:

    +

    Each SDK comes with its own implementation of the backend for the application. The + backend is a REST API available in the following languages:

    • Node.js
    • Java
      • @@ -33,25 +24,39 @@ interaction with data model only)

    Please refer to the travel-app documentation of your SDK of choice for specifics for that - particular SDK, like backend dependencies, source-code checkout and running instructions.

    -

    The application allows users to find flights by entering airports and travel dates. For the - airport entries, the app uses a N1QL query bound to an Angular type ahead directive to - suggest airport names based on the first few letters entered.

    + particular SDK, like backend dependencies, source-code checkout, and running instructions.

    +

    The travel app + front-end is the same in each SDK implementation and is built on an Angular2 stack.

    + +

    While the app runs, you can get a peek of what is happening in + the backend via the "Narration" — a console-like UI element that can display N1QL queries, for instance. + It is only visible in components that communicate with the server, but can also be collapsed. +

    + +

    The application allows users to find flights by entering airports and travel dates. For the + airport entries, the app uses a N1QL query bound to an Angular typeahead directive to + suggest airport names based on the first few letters entered.

    + +

    The focus of this tutorial is the backend. The documentation for the travel app illustrates the data model and + walks through the N1QL and FTS queries used to select flights and search for hotels.

    +
    Generic set up

    You'll need:

    • Your favorite editor or IDE
    • Your SDK of choice and its specific dependencies
      • -
    • The sample app source code for your SDK from GitHub (check the instructions and make sure to use the implementation compatible with the v2 API)
      • -
    • A local Couchbase 4.5 (or greater) installation (make sure that the - travel-sample bucket has been loaded from 4.5 and that there is, - at least, one node with data, query, index and full text search services in the cluster)
      • +
    • A local Couchbase 5.1 (or later) installation — make sure that the + travel-sample bucket has been loaded, and that there is + at least one node with data, query, index, and full text search services in the cluster
      • +
    • The sample app source code for your SDK from GitHub — check the instructions and make sure to use + the implementation compatible with the v2 API
    • That's it!

    @@ -60,93 +65,101 @@ advanced Multi-Dimensional Scaling (MDS) configurations are supported. It's just easier to start a development environment with components running locally.

    -

    Download Couchbase Server 4.6 - and install it. As you follow the download instructions and setup wizard, make sure you keep all the services - (data, query, and index) selected. Make sure also to install the - sample bucket named travel-sample - (introduced in Couchbase Server 4.0) because it contains the data used in this - tutorial.

    - It is important to install the - travel-sample bucket from a 4.5 or later version of - Couchbase Server, as it has been enriched with hotels data. If you have installed the sample bucket from an earlier +

    If you do not already have Couchbase Server 5.1 or later installed, + download Couchbase + Server and install it. When installed, open a browser and navigate to the Couchbase Web Console + URL — by default, 127.0.0.1:8091. When prompted, create a new cluster. Make sure all the + services (Data Service, Index Service, Search + Service and Query Service) are selected.

    + + It is important to install the + travel-sample bucket from Couchbase Server 5.1 or later, + as it has been enriched with hotels data. If you have installed the sample bucket from an earlier version, delete the bucket and follow the instructions below. -

    If you already have Couchbase Server 4.5 or later installed but did not install the - travel-sample bucket (or you installed it from a 4.0 version), open the Couchbase Web Console - and select Settings Sample Buckets . - Select the travel-sample checkbox, and then click Create. A notification - box in the upper-right corner disappears when the bucket is ready to use.

    -

    Finally, in order to be able to search for hotels, an FTS index called hotels must be created on - the travel-sample bucket. Go to Indexes Full Text New Full Text Index and create the hotels - index:.

    -

    Note that the index mapping could be better tuned, but a generic mapping will also work fine for this example.

    + +

    If you have not installed the travel-sample bucket from Couchbase Server 5.1, + select Settings Sample Buckets. + Select the travel-sample checkbox, and then click Load Sample Data. + The activity notification in the upper-right corner shows that the bucket is being added. + When the bucket is ready to use, it is listed under the Installed Samples heading.

    + +

    In order to be able to search for hotels, you must create an FTS index on the + travel-sample bucket. Select Search Add + Index and create an index named hotels. +

    + + The index mapping could be better tuned, but a generic mapping will also work fine for this example. + +

    Finally, download the complete source code for the backend, then build and run the Travel App. See the links above for instructions for your development environment.

    + +

    The app displays a message showing the login URL for the frontend. Make a note of this so you can log in.

    +
    Using the Travel App -

    To find a flight:

    +

    To find and book a flight:

    1. Open a browser and navigate to the login URL that was displayed when you started the app.
      2. -
    2. Sign in to Couchbase Travel by providing your credentials, or create a new account (which - will expire after some time depending on the backend used, usually 1 hour): Sign in to Couchbase Travel by creating a new account, or providing existing credentials. (New + accounts expire after some time depending on the backend used, usually 1 hour.)
      3. -
    3. Find a flight: -

      In the Airport or City section, enter an airport code or city name in the - From and To fields.

      4. +
    4. In the Airport or City section, enter an airport code or city name in the + From and To fields.
    5. In the Travel Dates section, enter Leave and Return dates in MM/DD/YYYY format.
    6. Click Find Flights. The app displays the available flights for the outbound and return legs of the trip.
      7. -
    7. Select one or more flights to add to your cart by clicking the choose button next +
    8. Select one or more flights to add to your cart by clicking the Choose button next to each flight.
    -

    Now you have flights stored locally in your cart. Go to the cart page and review your flights:

    +

    Now you have flights stored locally in your cart. You can see that the number of flights in the cart and the total cost are displayed in the navigation bar at the top right. To review your flights, navigate to the cart page:

      -
    1. Click on the cart icon button in the navigation bar, top right
      2. -
    2. See that the number of flights in the cart and the total cost are displayed in said navigation bar.
      3. -
    3. Review the flights in your cart, and book one by clicking on the book button next to it. +
    4. Click on the cart icon in the navigation bar. +
      5. +
    5. Review the flights in your cart, and book one by clicking on the Book button next to it.
      6. -
    6. The flight is now booked, as confirmed by a message on the page
    -

    The flights that you've booked are stored in the backend. You can see which flights you've booked by navigating - to the "user" page. The backend uses authentication to control access to the endpoints for booking and listing - flights.

    - -

    To search for an hotel, navigate to the hotel page:

    +

    The flight is now booked, as confirmed by a message on the page.

    +

    The flights that you've booked are stored in the backend. The backend uses authentication to control access to the endpoints for booking and listing flights.

    +

    To see which flights you've booked, navigate to the user page:

      -
    1. click on the "earth" icon in the navigation bar (rightmost icon).
      2. -
    2. click on the Find Hotels button if you want to have a look at any first 100 hotels.
      3. -
    3. Refine your search using the input boxes, either by keyword, geographically or both.
      4. +
    4. Click on the user icon in the navigation bar. + +
      5. +
    +

    To search for a hotel, navigate to the hotel page:

    +
      +
    1. Click on the hotel icon to the right of the navigation bar. +
      2. +
    2. Don't enter any search criteria yet. Just click on the Find Hotels button. The first 100 hotels are displayed.
      3. +
    3. Now enter a keyword in the Description box, or a location in the Location box, and click on Find Hotels again to refine your search.
    - You should make sure that you have installed Couchbase 4.5 with FTS enabled. The API backend that - is used by the hotels page makes use of an FTS index that you have to create, named hotels. + To use the hotel search, you must have created an FTS index named hotels, as described earlier. The API backend that is used by the hotel page makes use of this FTS index.
    The Travel App data model -

    The data model for the travel app uses several distinct document types: airline, route, airport and landmark. - In Couchbase Server 4.5 and later it has been enriched with a 5th type: hotel.

    +

    The data model for the travel app uses several distinct document types: airline, route, airport, and hotel.

    The model for each kind of document contains:

    • A key that acts as a primary key
    • An id field that identifies the document
    • A type field that identifies the kind of document

    -

    The following figure illustrates the relationship between the different kinds of documents in 4.0. It +

    The following figure illustrates the relationship between the different kinds of documents in version 5.1 of the Travel App. It shows the primary key, ID, and type fields that each document has, plus a few representative fields in each type of document.

    - Documents in the travel app 4.0 data model - - -

    Hotels, like landmarks, don't have relations to other types. The figure below displays essential information - about the hotel type introduced in the sample in 4.5:

    - - The hotel type in the travel app 4.5 data model - + Documents in the Travel App 5.1 data model + +

    Landmark documents are not used in the travel-sample application, which focuses on a more specific subset + of point of interests that have their own type: hotels.

    +

    Hotels, like landmarks, don't have relations to other types.

    @@ -157,8 +170,7 @@ and the airline call sign.

    For airline documents, the value of the type field is airline.

    - - airline_24 ←This is the key, which also acts as a primary key + airline_24 ←This is the key, which also acts as a primary key { "active": "Y", "callsign": "AMERICAN", @@ -182,8 +194,7 @@ the corresponding airline document.

    For route documents, the value of the type field is route.

    - - route_5966 ←This is the key, which also acts as a primary key + route_5966 ←This is the key, which also acts as a primary key { "id": "5966", "type": "route", ←This is the type identifier for the document @@ -208,7 +219,7 @@ Administration (FAA) location identifier.

    For airport documents, the value of the type field is airport.

    - airport_3577 ←This is the key, which also acts as a primary key + airport_3577 ←This is the key, which also acts as a primary key { "travel-sample": { "airportname": "Seattle Tacoma Intl", @@ -227,16 +238,6 @@ } }
    -
    - Landmark documents -

    Landmark documents contain details about points of interest such as hotels. They - include information such as name, location, price, contact information, and the kind - of activity that the point of interest provides.

    -

    For landmark documents, the value of the type field is - landmark.

    -

    Landmark documents are not used in the travel-sample application, which focuses on a more specific subset - of point of interests that have their own type: hotels.

    -
    Hotel documents @@ -244,8 +245,7 @@ such as a name, description, location (coutry, state, city and address), price and services, and even customer reviews.

    For hotel documents, the value of the type field is hotel.

    - - hotel_10180 ←This is the key, which also acts as a primary key + hotel_10180 ←This is the key, which also acts as a primary key { "address": "8301 Hollister Ave", "checkin": "12PM", @@ -294,7 +294,7 @@

    The application serves an HTML file named index.html (in an src/app/ folder usually contained in a root folder - dedicated to frontend files, eg. the public path in the Node.js + dedicated to frontend files, e.g. the public path in the Node.js application). This file contains references to various included script files for front side JavaScript components such as Bootstrap and jQuery. It also contains a small script snippet that bootstraps the Angular2 application, as well as an Angular loop @@ -323,7 +323,7 @@ The Application Services - +

    @@ -342,7 +342,7 @@ Home Component call hierarchy - +

    The application attempts to find an airport based on the codes used for the name (case sensitive), and codes for Federal Aviation @@ -467,7 +467,7 @@ SELECT * FROM default b JOIN default a ON KEYS b.joinField Hotels component call hierarchy - +

    The form is bound to the component using a FormBuilder, and it has two optional text inputs: a description and a location field. When submitting the form, @@ -475,7 +475,7 @@ SELECT * FROM default b JOIN default a ON KEYS b.joinField representation of the form's content. The findHotels method will use these values to determine if it should hit a more specific endpoint, in the case where a particular description keyword or location are requested by the form.

    -

    For instance, to search for an hotel in France with a description matching "golf" (in +

    For instance, to search for a hotel in France with a description matching "golf" (in a full text search sense, possibly including alternative spellings, words with the same root, etc...), one should use the /api/hotel/France/golf endpoint. The possible endpoints are:

    @@ -540,7 +540,7 @@ SELECT * FROM default b JOIN default a ON KEYS b.joinField landmark.
  • There are quite a few other FTS query types. For instance, a match query is much like a match phrase query, except it only considers one term to search for, and can - apply fuzzyness (eg. to match close words with a slightly different spelling).
    • + apply fuzzyness, e.g. to match close words with a slightly different spelling.

    Finally, we notice that at the same level as the query entry there is also a size attribute. This is one of the many tunings applicable to diff --git a/content/sdk/sdk-authentication.dita b/content/sdk/sdk-authentication.dita index d004f6b7a1..69aa430adf 100644 --- a/content/sdk/sdk-authentication.dita +++ b/content/sdk/sdk-authentication.dita @@ -111,8 +111,7 @@ cluster.authenticate(authenticator)

  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    diff --git a/content/sdk/sdk-user-management-overview.dita b/content/sdk/sdk-user-management-overview.dita index 38b5af3fda..af96050d72 100644 --- a/content/sdk/sdk-user-management-overview.dita +++ b/content/sdk/sdk-user-management-overview.dita @@ -219,8 +219,7 @@

  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    diff --git a/content/sdk/sdk-user-management.dita b/content/sdk/sdk-user-management.dita index 2238afa89e..6567213ea6 100644 --- a/content/sdk/sdk-user-management.dita +++ b/content/sdk/sdk-user-management.dita @@ -203,8 +203,7 @@

  • Roles required for resource-access, and the privileges they entail, see - RBAC - for Applications. + Roles.

    diff --git a/content/sdk/shared/images/field_level_encryption--architecture.png b/content/sdk/shared/images/field_level_encryption--architecture.png new file mode 100644 index 0000000000..65c495c152 Binary files /dev/null and b/content/sdk/shared/images/field_level_encryption--architecture.png differ diff --git a/content/sdk/shared/images/field_level_encryption--nuget.png b/content/sdk/shared/images/field_level_encryption--nuget.png new file mode 100644 index 0000000000..2e86a1de07 Binary files /dev/null and b/content/sdk/shared/images/field_level_encryption--nuget.png differ diff --git a/content/sdk/shared/images/icon-cart.png b/content/sdk/shared/images/icon-cart.png new file mode 100644 index 0000000000..d75dd840f0 Binary files /dev/null and b/content/sdk/shared/images/icon-cart.png differ diff --git a/content/sdk/shared/images/icon-home.png b/content/sdk/shared/images/icon-home.png new file mode 100644 index 0000000000..57c00cc991 Binary files /dev/null and b/content/sdk/shared/images/icon-home.png differ diff --git a/content/sdk/shared/images/icon-hotel.png b/content/sdk/shared/images/icon-hotel.png new file mode 100644 index 0000000000..3dbf6d1fd4 Binary files /dev/null and b/content/sdk/shared/images/icon-hotel.png differ diff --git a/content/sdk/shared/images/icon-user.png b/content/sdk/shared/images/icon-user.png new file mode 100644 index 0000000000..27e06f10f6 Binary files /dev/null and b/content/sdk/shared/images/icon-user.png differ diff --git a/content/sdk/shared/images/travel-app-data-model.png b/content/sdk/shared/images/travel-app-data-model.png index 02f6e18373..8a0be6da63 100644 Binary files a/content/sdk/shared/images/travel-app-data-model.png and b/content/sdk/shared/images/travel-app-data-model.png differ diff --git a/content/sdk/shared/images/travel-app-fts-index.png b/content/sdk/shared/images/travel-app-fts-index.png index e2c9522a75..1df2ec443c 100644 Binary files a/content/sdk/shared/images/travel-app-fts-index.png and b/content/sdk/shared/images/travel-app-fts-index.png differ diff --git a/content/sdk/shared/reusable/sdk-xattr-overview.dita b/content/sdk/shared/reusable/sdk-xattr-overview.dita index 8d9ebbc756..6e6f55f62d 100644 --- a/content/sdk/shared/reusable/sdk-xattr-overview.dita +++ b/content/sdk/shared/reusable/sdk-xattr-overview.dita @@ -44,8 +44,8 @@

    For more information, see - Extended Attributes — Fundamentals. + Extended Attributes.

    • diff --git a/content/sdk/shared/tracing.dita b/content/sdk/shared/tracing.dita new file mode 100644 index 0000000000..f582ccb58a --- /dev/null +++ b/content/sdk/shared/tracing.dita @@ -0,0 +1,121 @@ + + + + Threshold Logging Tracing + By collecting and analyzing each operation an SDK performs, + information can be made available to identify problem areas when an operation takes + an unusually long time to complete. For example pin-pointing whether any delay lies with + network issues, rather than a Couchbase Server operation, or if most delays are concentrated + on a particular Server node. (Response Time Observability, a.k.a. Tracing, is Enterprise Edition + only - not available in the Community Edition.) + + Threshold Logging Tracing is part of the SDK, and will work with both the + Community and Enterprise Editions of Couchbase Server. However, Data Service operation duration is only + available in the Enterprise Edition. + +
    + Why Tracing? +

    Tracing is recording details about discrete steps or phases of a request lifecycle, + such as request encoding / decoding, or dispatching to server. These phases are timed + independently, and contain additional contextual information. A single request is typically + made up of multiple tracing points.

    +
    + +
    + Open Tracing +

    OpenTracing is a + standardised API to structure tracing information in a consistent and predictable manner. +

    +
    + +
    + Threshold Configuration Settings +

    The threshold tracer receives completed spans and verifies if an operation has exceeded the + given threshold for the operation type. Operations that exceed the threshold are periodically + logged with a total count and a sample of the slowest ones. The following are SDK configuration properties.

    + + Threshold Logging Tracer Properties + + + + + + + Property Name + Description + Default Value + + + + + OperationTracingEnabled + Boolean used to determine tracing is enabled. Defaults to using the + ThesholdLoggingTracer if enabled. When false a Noop or similar tracing + implementation should be used instead. + true + + + OperationTracingServerDurationEnabled + Boolean used to instruct the SDK to try and retrieve duration metrics from the server for KV operations. + true + + + ThresholdLoggingTracerInterval + The interval between executions that process the collected operation spans. + Expressed in milliseconds. + + 10,000 (10 seconds) + + + ThresholdLoggingTracerSampleSize + The maximum number of items to log per service. + 10 + + + ThresholdLoggingTracerKVThreshold + The KV operation operation threshold. Expressed in microseconds. + 500,000 (500 milliseconds) + + + ThresholdLoggingTracerViewsThreshold + The View query operation threshold. Expressed in microseconds. + 1,000,000 (1 second) + + + ThresholdLoggingTracerQueryThreshold + The N1QL query operation threshold. Expressed in microseconds. + 1,000,000 (1 second) + + + ThresholdLoggingTracerSearchThreshold + The FTS query operation threshold. Expressed in microseconds. + 1,000,000 (1 second) + + + ThresholdLoggingTracerAnalyticsThreshold + The Analytics query operation threshold. Expressed in microseconds. + 1,000,000 (1 second) + + + OrphanedResponseLoggingEnabled + Boolean used to determine if orphaned response logging is enabled. + true + + + OrphanedResponseLoggingInterval + The interval used to flush orphaned response information to the log. Expressed in microseconds. + 10,000 (10 seconds) + + + OrphanedResponseLoggingSampleSize + The number of sample orphaned responses whose to log additional information for per execution. + 10 + + + +
    + + +
    + +     diff --git a/content/sdk/shared/views.dita b/content/sdk/shared/views.dita index fc4dadcbb3..4d70b9db00 100644 --- a/content/sdk/shared/views.dita +++ b/content/sdk/shared/views.dita @@ -14,8 +14,7 @@ and written in JavaScript.

    MapReduce queries can be further customized during query time to allow only a subset (or range) of the data to be returned.

    - See the and - sections of the general documentation to learn more about views and their architecture. + See Views.
    diff --git a/content/sdk/subdocument-operations.dita b/content/sdk/subdocument-operations.dita index 9dee874192..725eb05280 100644 --- a/content/sdk/subdocument-operations.dita +++ b/content/sdk/subdocument-operations.dita @@ -17,7 +17,7 @@
    Sub-documents

    Starting with Couchbase Server 4.5 you can atomically and efficiently update and - retrieve parts of a document. These parts are called sub documents. + retrieve parts of a document. These parts are called sub-documents. While full-document retrievals retrieve the entire document and full document updates require sending the entire document, sub-document retrievals only retrieve relevant parts of a document and sub-document updates only require sending the @@ -33,9 +33,10 @@

    In order to use sub-document operations you need to specify a path indicating - the location of the subdocument. The path follows N1QL syntax. Considering - the - document{ + the location of the sub-document. The path follows N1QL syntax (see below, + and ). Considering + the document: { "name": "Douglas Reynholm", "email": "douglas@reynholmindustries.com", "addresses": { @@ -58,14 +59,14 @@ 157, 42, 999 ] } -}the +}The paths name, addresses.billing.country and purchases.complete[0] are all valid paths.

    RetrievingThe lookup-in operations queries the document for a certain path(s) and returns that/those path(s). You have a choice of actually retrieving - the document path using the subdoc-get subdocument operation, or simply querying - the existence of the path using the subdoc-exists subdocument operation. The + the document path using the subdoc-get sub-document operation, or simply querying + the existence of the path using the subdoc-exists sub-document operation. The latter saves even more bandwidth by not retrieving the contents of the path if it is not needed.import couchbase.subdocument as SD # Use SD alias for brevity rv = bucket.lookup_in('customer123', SD.get('addresses.delivery.country')) @@ -89,17 +90,17 @@ echo $frags->value[1]['value'] . ", Code=" . $frags->value[1]['code'] . "\n"; document. The simplest of these operations is subdoc-upsert, which, just like the fulldoc-level upsert, this will either modify the value of an existing path or create it if it does not - exist:

    bucket.mutate_in('customer123', SD.upsert('fax', '775-867-5309'))Likewise, + exist:

    bucket.mutate_in('customer123', SD.upsert('fax', '311-555-0151'))

    Likewise, the subdoc-insert operation will only add the new value to the path if it does not - existbucket.mutate_in('customer123', SD.insert('purchases.complete', [42, True, None])) -# SubdocPathExistsErrorDictionary + exist:

    bucket.mutate_in('customer123', SD.insert('purchases.complete', [42, True, None])) +# SubdocPathExistsError

    Dictionary values can also be replaced or removed, and you may combine any number of mutation operations within the same general mutate-in API. Here's an example of one which replaces one path and removes - another.bucket.mutate_in('customer123', + another.

    bucket.mutate_in('customer123', SD.remove('addresses.billing'), - SD.replace('email', 'doug96@hotmail.com'))
    + SD.replace('email', 'dougr96@hotmail.com'))
    Array append and prepend

    The subdoc-array-prepend and subdoc-array-append operations are true array prepend and append operations. Unlike fulldoc append/prepend operations (which simply @@ -140,25 +141,25 @@ bucket.mutate_in('my_array', SD.array_append('', 'some element')) arrays like unique sets, using the subdoc-array-addunique command. This will do a check to determine if the given value exists or not before actually adding the item to the - array

    bucket.mutate_in('customer123', SD.push_unique('purchases.complete', 95)) + array:

    bucket.mutate_in('customer123', SD.push_unique('purchases.complete', 95)) # => Success bucket.mutate_in('customer123', SD.push_unique('purchases.abandoned', 42)) -# => SubdocPathExists exception!Note +# => SubdocPathExists exception!

    Note that currently the addunique will fail with a Path Mismatch error if the array contains JSON floats, objects, or arrays. The addunique operation will also fail with Cannot Insert if the value to - be added is one of those types as well.

    Note that the actual position of the new + be added is one of those types as well.

    Note that the actual position of the new element is undefined, and that the array is not ordered.

    -
    Array insertionNew elements can also be inserted into an +
    Array insertion

    New elements can also be inserted into an array. While append will place a new item at the end of an array and prepend will place it at the beginning, insert allows an element to be inserted at a specific position. The position is indicated by the last path component, which should be an array index. For example, to insert "cruel" as the second element in the array ["Hello", "world"], the code would look - like:bucket.mutate_in('array', SD.arrayinsert('[1]', 'cruel'))Note + like:

    bucket.mutate_in('array', SD.arrayinsert('[1]', 'cruel'))

    Note that the array must already exist and that the index must be valid (i.e. it must not - point to an element which is out of bounds).

    + point to an element which is out of bounds).

    Counters and numeric fields

    Counter operations allow the manipulation of a numeric value inside a document. These operations are logically similar to the counter operation on an entire @@ -169,10 +170,10 @@ cur_count = rv[0] # => 1The value.

    bucket.upsert('player432', {'gold': 1000}) rv = bucket.mutate_in('player432', SD.counter('gold', -150)) print('player432 now has {0} gold remaining'.format(rv[0])) -# => player 432 now has 850 gold remainingThe +# => player 432 now has 850 gold remaining

    The existing value for subdoc-counter operations must be within range of a 64 bit signed integer. If the value does not exist, the subdoc-counter operation will - create it (and its parents, if create-parents is enabled).

    Note that there are + create it (and its parents, if create-parents is enabled).

    Note that there are several differences between subdoc-counter and the full-document counter operations:

    • Sub-document counters have a range of -9223372036854775807 to @@ -190,7 +191,7 @@ print('player432 now has {0} gold remaining'.format(rv[0]))

    Executing multiple operations -

    Multiple subdocument operations can be executed at once on the same document, +

    Multiple sub-document operations can be executed at once on the same document, allowing you to retrieve or modify several sub-documents at once. When multiple operations are submitted within the context of a single lookup-in or mutate-in command, the server will execute all the operations with the @@ -241,7 +242,7 @@ print('player432 now has {0} gold remaining'.format(rv[0])) create-parents or create-intermediates option may be used.bucket.mutate_in('customer123', SD.upsert('level_0.level_1.foo.bar.phone', - {'num': '775-867-5309', 'ext': 16}, + {'num': '311-555-0101', 'ext': 16}, create_parents=True))

    CAS Semantics

    Subdoc mostly eliminates the need for tracking the @@ -249,11 +250,11 @@ print('player432 now has {0} gold remaining'.format(rv[0])) value. Subdoc operations are atomic and therefore if two different threads access two different sub-documents then no conflict will arise. For example the following two blocks can execute concurrently without any risk of - conflictbucket.mutate_in('customer123', SD.array_append('purchases.complete', 999)) - bucket.mutate_in('customer123', SD.array_append('purchases.abandoned', 998))

    Even + conflict:bucket.mutate_in('customer123', SD.array_append('purchases.complete', 999)) + bucket.mutate_in('customer123', SD.array_append('purchases.abandoned', 998))

    Even when modifying the same part of the document, operations will not necessarily - conflict, for example two concurrent subdoc-array-append to the same array will - both succeed, never overwriting the other.

    While CAS is no longer required to ensure + conflict. For example, two concurrent subdoc-array-append operations to the same array will + both succeed, never overwriting the other.

    While CAS is no longer required to ensure document updates are preserved, it may still be needed to ensure document state remains consistent over multiple invocations of mutate-in: Sometimes it's important to ensure the entire document didn't change state since the last @@ -274,7 +275,7 @@ print('player432 now has {0} gold remaining'.format(rv[0]))

  • Path mismatch: This means the path may exist in the document, but that there is a type conflict between the path in the document and the path in the command. Consider the - document,{ "tags": ["reno", "nevada", "west", "sierra"] }The + document:{ "tags": ["reno", "nevada", "west", "sierra"] }The path tags.sierra is a mismatch, since tags is actually an array, while the path assumes it is a JSON object (dictionary).
    • @@ -284,7 +285,7 @@ print('player432 now has {0} gold remaining'.format(rv[0])) commands such as subdoc-array-insert expect array elements as their final component, while others such as subdoc-upsert and subdoc-insert expect dictionary (object) keys. - Because subdocument operations are executed using either mutate-in or + Because sub-document operations are executed using either mutate-in or replace-in, if a command fails a top-level error is reported (Multi Command Failure), rather than an individual error code (e.g. Path Not Found). When receiving a top-level error code, you should traverse the @@ -329,5 +330,9 @@ print escape_component("[\"mixed\\") # `[\"mixed\\` Currently, paths cannot exceed 1024 characters, and cannot be more than 32 levels deep.
    +
    + XDCR +

    XDCR only replicates full documents. Sub-documents are only replicated as part of the full document.

    +
    diff --git a/content/sdk/webui-cli-access.dita b/content/sdk/webui-cli-access.dita index 484df57053..1b66c2df10 100644 --- a/content/sdk/webui-cli-access.dita +++ b/content/sdk/webui-cli-access.dita @@ -24,8 +24,8 @@ document lookups by their IDs (or keys) and the cbq tool to issue N1QL queries. This section will discuss the installation and usage of the - cbc tool, and is targeted towards Linux and Mac OS X users (though - cbc can also be used on Windows). + cbc tool, and is targeted towards Linux and Mac OS X users. + You can also use cbc on Windows by extracting the archive with libcouchbase. The archive contains cbc.exe by default.
    @@ -118,7 +118,7 @@ cbq> select airportname FROM `travel-sample` WHERE airportname IS NOT NULL LI

    - You can use the Couchbase Web Console to view, edit, and create JSON documents up to 2.5KB in + You can use the Couchbase Web Console to view, edit, and create JSON documents up to 256KB in size. To access documents using the Couchbase Web Console:

      diff --git a/content/security/concepts-rba-for-apps.dita b/content/security/concepts-rba-for-apps.dita index f9ff4098bc..0d9a885188 100644 --- a/content/security/concepts-rba-for-apps.dita +++ b/content/security/concepts-rba-for-apps.dita @@ -3,15 +3,12 @@ - RBAC for Applications + Roles - Building on the Role-Based Access Control (RBAC) security model introduced in 4.5 for - administrators, Couchbase Server 5.0 adds RBAC for applications; so that you can control, - at both broad and granular levels, what end-users and application services can do. Thus, you - can limit access to just the information they need, nothing more or less, and meet - compliance requirements. + A Couchbase role permits one or more resources to be accessed according + to defined privileges. @@ -25,36 +22,663 @@ Couchbase roles each have a fixed association with a set of one or more privileges. Each privilege is associated with a resource. Privileges are actions such as Read, Write, - Execute, Manage, Flush, or + Execute, Manage, Flush, and List; or a combination of some or all of these.

      - When an application attempts to access a resource, the application's roles and privileges are checked - by Couchbase Server. If the assigned roles contain privileges that support the kind + Roles are of two kinds: +

      + +

      + +

      + +
        +
      • + Administration and Global: Associated with cluster-wide privileges. Some of these roles are for + administrators; who might manage cluster-configurations; or read statistics; or enforce security. Others are for + users and user-defined applications that require access to specific, cluster-wide resources. +

        + +

        +
        • + +
      • + Per Bucket: Associated with one or more buckets. These roles support the reading and writing of + bucket-settings; the management of services, indexes, and replication procedures; and access to data. +

        + +

        +
        • + +
      + +

      + When an administrator, user, or application attempts to access a resource, they must authenticate, by means + of a username and password. The roles + and privileges associated with these credentials are checked + by Couchbase Server. If the associated roles contain privileges that support the kind of access that is being attempted, access is granted; otherwise, it is denied.

      - The following list contains all application-roles supported by Couchbase RBAC. Each role explained - by means of a description and a table: the table lists the privileges in + The following list contains all roles supported by Couchbase Server, Enterprise Edition. Each role is + explained + by means of a description and (in most cases) a table: the table lists the privileges in association with - resources. Where a privilege is associated with a resource, + resources. The header of each table states the role's name, followed by its alias name + in parentheses: alias names are used in commands and queries. + In each table-body, where a privilege is associated with a resource, this is indicated with a check-mark. Where a privilege is not associated with a resource (or where association would not be applicable), this is indicated with - a cross. + a cross. Resources not referred to in a particular table have no privileges + associated with them in the context of role being described.

      + +

      + Note that some roles grant access to Couchbase Web Console; while others do not. The set of features + displayed within the console varies, according to role. +

    + +
    + + Full Admin + + +

    + The Full Admin role (an Administration and Global role) supports full access to all Couchbase-Server + features and resources, including those of security. The role allows access + to the Couchbase Web Console, and allows the reading and writing of bucket-data. +

    + +

    + This role is also available in Couchbase Server Community Edition. +

    + +
    + +
    + + Read-Only Admin + + +

    + The Read-Only Admin role (an Administration and Global role) supports the reading of Couchbase Server-statistics: + this includes registered usernames with roles and authentication domains, but excludes + passwords. + The role allows access to Couchbase Web Console. +

    + +

    + This role is also available in Couchbase Server Community Edition. +

    + + + + + + + + + + + + + + + + + Role: Read-Only Admin (ro_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Cluster + + + + + + + + UI (except Passwords) + + + + + + + + Security (except Passwords) + + + + + + + + Bucket Data + + + + + + + + +
    + +
    + + + +
    + + Cluster Admin + + +

    + The Cluster Admin role (an Administration and Global role) allows the management of all cluster features + except security. + The role allows access to Couchbase Web Console, but does not permit the + writing of data. +

    + + + + + + + + + + + + + + + Role: Cluster Admin (cluster_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Cluster (except Passwords) + + + + + + + + UI (except Passwords) + + + + + + + + Security (except Passwords) + + + + + + + + Bucket Data + + + + + + + + +
    + +
    + + +
    + + Bucket Admin + + +

    + The Bucket Admin role (an Administration and Global role) allows the management of all features + for a bucket (including starting and stopping XDCR). + The role allows access to Couchbase Web Console, but does not permit the + reading or writing of data. +

    + + + + + + + + + + + + + + + + Role: Bucket Admin (bucket_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Cluster + + + + + + + + Bucket (including XDCR) + + + + + + + + Bucket Data + + + + + + + + Bucket UI + + + + + + + + Other UI + + + + + + + + +
    + +
    + + +
    + + Views Admin + + +

    + The Views Admin role (an Administration and Global role) allows the reading of views within a bucket. + The role allows access to Couchbase Web Console. +

    + + + + + + + + + + + + + + + + Role: Views Admin (views_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + + Bucket Data (Views) + + + + + + + + Bucket Data (Other) + + + + + + + + Bucket Settings + + + + + + + + UI (Views) + + + + + + + + UI (Other) + + + + + + + + +
    + +
    + +
    + + XDCR Admin + + +

    + The XDCR Admin role (an Administration and Global role) allows use of XDCR features, to create + cluster references and + replication streams. + The role allows access to Couchbase Web Console. +

    + + + + + + + + + + + + + + + + Role: XDCR Admin (replication_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + XDCR for Cluster and Bucket + + + + + + + + Bucket Data + + + + + + + + Bucket Settings + + + + + + + + UI (XDCR) + + + + + + + + UI (Other) + + + + + + + + +
    + +
    + + + + +
    + + Search Admin + + +

    + The Search Admin role allows management of all features of + the Search Service, per bucket. + The role allows access to Couchbase Web Console. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred to as + FTS Admin. +

    + + + + + + + + + + + + + + + + Role: Search Admin (fts_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + + Bucket Data (Search) + + + + + + + + Bucket Data (Other) + + + + + + + + Bucket Settings + + + + + + + + UI (Search) + + + + + + + + UI (Other) + + + + + + + + Services and Curl + + + + + + + + Pools + + + + + + + + +
    + +
    + + + + + + +
    - Bucket Full Access + Application Access

    - The Bucket Full Access role provides full access to bucket data. Note that this privilege is available for the Community Edition of - Couchbase Server, as well as for Enterprise Edition. + The Application Access role provides read and write access to data, per bucket. + The role does not allow access to Couchbase Web Console: it is intended for + applications, rather than users. + Note that this role is also available in the Community Edition of + Couchbase Server.

    @@ -63,23 +687,25 @@ accessed by specifying bucket-name and bucket-password: however, bucket-passwords are not recognized by Couchbase Server 5.0 and after. Therefore, for each pre-existing bucket, - the 5.0 upgrade-process creates a new user, + the upgrade-process for 5.0 and after creates a new user, whose username is identical to the bucket-name; and whose password is identical to the former bucket-password, if one existed. If no bucket-password existed, the user is created with no password. This migration-process allows the same name-combination as before to be used in authentication. To ensure backwards compatibility, each system-created user is - assigned the Bucket Full Access role, which authorizes the same read-write + assigned the Application Access role, which authorizes the same read-write access to bucket-data as was granted before 5.0.

    - Use of the Bucket Full Access role is deprecated for buckets + Use of the Application Access role is deprecated for buckets created on Couchbase Server 5.0 and after: use the other bucket-access roles provided. + Note that in versions of Couchbase Server prior to 5.5, this role was referred to as + Bucket Full Access.

    - The below tables list each bucket's name followed by its alias name - in parenthesis. The alias names are used in commands and - are accessible only by N1QL queries. + + + @@ -97,7 +723,7 @@ - Role: Bucket Full Access (bucket_sasl) + Role: Application Access (bucket_full_access) @@ -118,7 +744,7 @@ - Bucket [ * | bucket-name ]: Data + Bucket Data @@ -128,7 +754,7 @@ - Bucket [ * | bucket-name ]: Views + Bucket Views @@ -155,7 +781,7 @@ - Bucket: [ * | bucket-name ] + Bucket @@ -184,20 +810,17 @@ - - - - - -
    Data Reader

    - The Data Reader role allows data to be read from a specified bucket. Note - that the role does not permit the running of N1QL queries (such as SELECT) against data. + The Data Reader role allows data to be read, per bucket. Note + that the role does not permit the running of N1QL + queries (such as SELECT) + against data. The role does not allow access to Couchbase Web Console: it is + intended to support applications, rather than users.

    @@ -235,7 +858,7 @@ - Bucket [ * | bucket-name ]: Docs + Bucket Docs @@ -243,7 +866,7 @@ - Bucket [ * | bucket-name ]: Meta + Bucket : Meta @@ -251,7 +874,7 @@ - Bucket [ * | bucket-name ]: Xattr + Bucket : Xattr @@ -283,7 +906,9 @@

    - The Data Writer role allows information to be written to and read from a specified bucket. + The Data Writer role allows data to be written, per bucket. The role does not allow access to + Couchbase Web Console: it is + intended to support applications, rather than users.

    @@ -301,7 +926,7 @@ - Role: Data Writer (data_reader_writer) + Role: Data Writer (data_writer) @@ -321,16 +946,16 @@ - Bucket [ * | bucket-name ]: Docs - + Bucket : Docs + - Bucket [ * | bucket-name ]: Xattr - + Bucket : Xattr + @@ -361,7 +986,10 @@

    - The Data DCP Reader role allows DCP streams to be read. + The Data DCP Reader role allows DCP streams to be initiated, per bucket. The + role does not allow access to Couchbase Web Console: it + is intended to support applications, rather than users. The role does + allow the reading of data.

    @@ -399,7 +1027,7 @@ - Bucket: [ * | bucket-name ]: Docs + Bucket: : Docs @@ -407,7 +1035,7 @@ - Bucket: [ * | bucket-name ]: Meta + Bucket: : Meta @@ -415,7 +1043,7 @@ - Bucket: [ * | bucket-name ]: DCP + Bucket: : DCP @@ -423,7 +1051,7 @@ - Bucket: [ * | bucket-name ]: Sxattr + Bucket: : Sxattr @@ -431,7 +1059,7 @@ - Bucket: [ * | bucket-name ]: Xattr + Bucket: : Xattr @@ -470,7 +1098,10 @@

    - The Data Backup role allows data to be backed up and restored. + The Data Backup role allows data to be backed up and restored, per bucket. The + role supports the reading of data. The role does not allow + access to Couchbase Web Console: it is intended to support applications, + rather than users.

    @@ -506,7 +1137,7 @@ - Bucket: [ * | bucket-name ]: Data + Bucket: : Data @@ -514,7 +1145,7 @@ - Bucket: [ * | bucket-name ]: Views + Bucket: : Views @@ -522,7 +1153,7 @@ - Bucket: [ * | bucket-name ]: FTS + Bucket: : FTS @@ -530,7 +1161,7 @@ - Bucket: [ * | bucket-name ]: Stats + Bucket: : Stats @@ -538,7 +1169,7 @@ - Bucket: [ * | bucket-name ]: Settings + Bucket: : Settings @@ -546,7 +1177,7 @@ - Bucket: [ * | bucket-name ]: Pools + Bucket: : Pools @@ -565,11 +1196,19 @@
    - Data Monitoring + Data Monitor

    - The Data Monitoring role allows all bucket-statistics to be read. + The Data Monitor role allows statistics to be read, per bucket. It does + not allow access to Couchbase Web Console, and does + not permit the reading of data. This role is intended to support + application-access, rather than user-access. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred + to as Data Monitoring.

    @@ -587,7 +1226,7 @@ - Role: Data Monitoring (data_monitoring) + Role: Data Monitor (data_monitoring) @@ -607,7 +1246,7 @@ - Bucket [ * | bucket-name ]: Stats + Bucket : Stats @@ -631,17 +1270,365 @@

    + +
    + + XDCR Inbound + + +

    + The XDCR Inbound role allows the creation of inbound XDCR streams, per bucket. It + does not allow access to Couchbase Web Console, and does not + permit the reading of data. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred to as + Replication Target. +

    + +
    + + + + + + + + + + + + + + + Role: XDCR Inbound (replication_target) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Settings + + + + + + + + Bucket : Meta + + + + + + + + Bucket : Stats + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Analytics Manager + + +

    + The Analytics Manager role allows management of Analytics, per bucket. + It also allows management of shadow datasets, provided that Data Read + permission has been granted on the corresponding buckets. This role allows access to Couchbase + Web Console. +

    + + + + + + + + + + + + + + + + + Role: Analytics Manager (analytics_manager) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Analytics + + + + + + + + Bucket : UI + + + + + + + + Bucket : Stats + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Analytics Reader + + +

    + The Analytics Reader role (an Administration and Global role) allows + querying of shadow data-sets. This + is defined as a global role because as multiple buckets may be combined into a single + shadow dataset. + The role allows access to Couchbase Web Console, and permits the reading + of data. +

    + + + + + + + + + + + + + + + + + Role: Analytics Reader (analytics_reader) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Analytics + + + + + + + + Bucket : UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Views Reader + + +

    + The Views Reader role allows data to be read from the views, + per bucket. This role does not allow access to Couchbase Web + Console, and is intended to support applications, rather than + users. +

    + + + + + + + + + + + + + + + + + Role: Views Reader (views_reader) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Docs + + + + + + + + Bucket : Views + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    - FTS Searcher + Search Reader

    - The role FTS Searcher allows Full Text Search indexes to be - searched by users with appropriate bucket-privileges. + The role Search Reader allows Full Text Search indexes to be + searched, per bucket. The role allows + access to Couchbase Web Console, and supports the reading of data. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred to as + FTS Searcher.

    @@ -659,7 +1646,7 @@ - Role: FTS Searcher (fts_searcher) + Role: Search Reader (fts_searcher) @@ -679,7 +1666,7 @@ - Bucket [ * | bucket-name ]: FTS + Bucket : FTS @@ -726,8 +1713,9 @@

    - The Query Select role allows the SELECT statement to be executed - on a specified bucket. + The Query Select role allows the SELECT statement to be executed, + perbucket. This role allows access to Couchbase Web Console, + but does not support the reading of data.

    @@ -765,7 +1753,7 @@ - Bucket [ * | bucket-name ]: N1QL, SELECT + Bucket : N1QL, SELECT @@ -797,6 +1785,85 @@

    + +
    + + Query Update + + +

    + The Query Update role allows the UPDATE statement to be executed, + per bucket. The role supports access to Couchbase Web Console, + but does not allow the reading of data. +

    + +
    + + + + + + + + + + + + + + + Role: Query Update (query_update) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : N1QL, UPDATE + + + + + + + + UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    @@ -804,8 +1871,9 @@

    - The Query Insert role allows the INSERT statement to be executed - on a specified bucket. + The Query Insert role allows the INSERT statement to be executed, + per bucket. The role supports access to Couchbase Web Console, + but does not allow the reading of data.

    @@ -843,7 +1911,7 @@ - Bucket [ * | bucket-name ]: N1QL, INSERT + Bucket : N1QL, INSERT @@ -882,8 +1950,9 @@

    - The Query Delete role allows the DELETE statement to be executed - on a specified bucket. + The Query Delete role allows the DELETE statement to be executed, + per bucket. The role supports access to Couchbase Web + Console, but does not allow the reading of data.

    @@ -921,7 +1990,7 @@ - Bucket [ * | bucket-name ]: N1QL, DELETE + Bucket : N1QL, DELETE @@ -961,8 +2030,9 @@

    - The Query Manage Index role allows indexes to be managed for - a specified bucket. + The Query Manage Index role allows indexes to be managed, + per bucket. The role allows access to Couchbase Web Console, but does + not permit the reading of data.

    @@ -1000,7 +2070,7 @@ - Bucket [ * | bucket-name ]: N1QL, INDEX + Bucket : N1QL, INDEX @@ -1039,10 +2109,13 @@

    - The Query System Catalog role allows information to be looked up - in the system catalog: this includes system:indexes, system:prepareds, and + The Query System Catalog role (an Administration and Global role) allows information to be looked up + by means of N1QL + in the system catalog: this includes system:indexes, + system:prepareds, and tables listing current and past queries. This role is designed for troubleshooters, - who need to debug queries. + who need to debug queries. The role allows access to Couchbase Web Console, but + does not permit the reading of bucket-items.

    @@ -1082,7 +2155,7 @@ - Bucket [ * | bucket-name ]: N1QL, INDEX + Bucket : N1QL, INDEX @@ -1091,7 +2164,7 @@ - Bucket [ * | bucket-name ]: N1QL, Meta + Bucket : N1QL, Meta @@ -1129,18 +2202,20 @@
    - Query External Access + Query Curl Access

    - The Query External Access role allows the N1QL curl function - to be executed by an externally authenticated user. + The Query Curl Access role (an Administration and Global role) allows the N1QL CURL function + to be executed by an externally authenticated user. The user + can access Couchbase Web Console, but cannot read data, other + than that returned by the N1QL CURL function.

    - Note that the Query External Access role should be assigned with caution, since + Note that the Query Curl Access role should be assigned with caution, since it entails risk: CURL runs within the local Couchbase Server - network; therefore, the assignee of the Query External Access + network; therefore, the assignee of the Query Curl Access role is permitted to run GET and POST requests on the internal network, while being themselves externally located. @@ -1151,6 +2226,11 @@ CURL Function.

    +

    + In versions of Couchbase Server prior to 5.5, this role was referred to as + Query External Access. +

    +
    @@ -1166,7 +2246,7 @@ - Role: Query External Access (query_external_access) + Role: Query Curl Access (query_external_access) @@ -1186,7 +2266,7 @@ - Bucket [ * | bucket-name ]: N1QL, curl + Bucket : N1QL, curl @@ -1214,13 +2294,16 @@
    -
    System Keyspaces (Tables)

    In Couchbase Server - 5.0, three new system keyspaces have been added:

      -
    • system:applicable_roles
      • -
    • system:my_user_info
      • -
    • system:user_info
      • -
    Along with these three keyspaces, meta data related to roles and user access has been - added as well.

    This brings the total number of system keyspaces up to 12:

    +

    + + System Keyspaces (Tables) + + +

    + The following system keyspaces are provided: +

    + +

    diff --git a/content/security/concepts-rba.dita b/content/security/concepts-rba.dita index da1875d36a..97753d1bc1 100644 --- a/content/security/concepts-rba.dita +++ b/content/security/concepts-rba.dita @@ -1,27 +1,49 @@ - RBAC for Administrators - Role-Based Access Control (RBAC) for administrators is designed to manage administrative + + + RBAC for Administrators + + + + Role-Based Access Control (RBAC) for administrators is designed to manage administrative access to the Couchbase system and to achieve segregation of administrative duties. With RBAC, each administrator can perform only a well-defined fixed set of operations based on the job - description. + description. + + -

    RBAC can be configured for internal or external users, and a userid can be mapped by the Full + +

    + RBAC can be configured for internal or external users, and a userid can be mapped by the Full Administrator to one or more fixed administrative roles. These roles contain a defined set of duties that can be performed in the system: users have no other administrative access - except for the one defined in their role-assignment.

    + except for the one defined in their role-assignment. +

    -

    Full administrators in Couchbase can manage user roles using the Couchbase CLI tools (as - described in ) or REST API (as described in - ).

    +

    + Full administrators in Couchbase can manage user roles using the Couchbase CLI tools (as + described in + admin-role-manage) or REST API (as described in + Role Based Admin Access (RBAC)). +

    - Fixed Administrative Roles in Couchbase

    Fixed administrative + + Fixed Administrative Roles in Couchbase + +

    + Fixed administrative roles in Couchbase Server are: Full, Cluster, Bucket, View, Replication, and FTS Administrator. -

    At a high-level, Couchbase has many resources:

      +

      + +

      + At a high-level, Couchbase has many resources: +

      + +
      • Cluster
      • Server
      • Bucket
        • @@ -40,7 +62,23 @@ Couchbase resources, including the security settings such as adding a user to, or removing a user from a particular fixed role. -
        +
        + +
        + +
        Security Administrator
        +
        The Security Administrator role can view all cluster statistics, and + can manage user roles; but + can neither grant the Full Administrator and Security Administrator + roles; nor + alter their own role. The role allows access to Couchbase Web Console, but does not + allow the reading of bucket-data
        +         +
        + + + +
        Read-only Administrator
        @@ -76,9 +114,8 @@
        Replication (XDCR) Administrator
        -
        The Replication Administrator role can now configure XDCR topology and - manage replications for High Availability without sharing administrative credentials - assigned to this role.
        +
        The Replication Administrator role can configure XDCR topology, and + manage replications for High Availability.
        @@ -144,7 +181,7 @@ W, R, S - ---------View (via REST API #8093) + ---------View (via REST API #8092) W, R, S W, R, S @@ -192,7 +229,7 @@ R - ---------View (via REST API #8093) + ---------View (via REST API #8092) R R @@ -241,7 +278,7 @@ W, R - ---------View (via REST API #8093) + ---------View (via REST API #8092) W, R W, R @@ -290,7 +327,7 @@ none - ---------View (via REST API #8093) + ---------View (via REST API #8092) W, R W, R @@ -339,7 +376,7 @@ none - ---------View via REST API #8093 + ---------View via REST API #8092 W, R W, R @@ -434,6 +471,19 @@
    +
    + + RBAC Logging + + +

    + Any authentication-failure will be logged in the log file for the resource on which access was attempted. + See + Access Logs for more information. +

    + +
    +     diff --git a/content/security/pict/addNewUserDialogRolesPanel.png b/content/security/pict/addNewUserDialogRolesPanel.png index f64cf60393..726d579834 100644 Binary files a/content/security/pict/addNewUserDialogRolesPanel.png and b/content/security/pict/addNewUserDialogRolesPanel.png differ diff --git a/content/security/pict/allBucketsCheckboxes.png b/content/security/pict/allBucketsCheckboxes.png new file mode 100644 index 0000000000..445f792f34 Binary files /dev/null and b/content/security/pict/allBucketsCheckboxes.png differ diff --git a/content/security/pict/auditView.png b/content/security/pict/auditView.png index 64ba6462f1..fd912f6c22 100644 Binary files a/content/security/pict/auditView.png and b/content/security/pict/auditView.png differ diff --git a/content/security/pict/couchbaseLogin.png b/content/security/pict/couchbaseLogin.png index da6eb564fa..6577398f85 100644 Binary files a/content/security/pict/couchbaseLogin.png and b/content/security/pict/couchbaseLogin.png differ diff --git a/content/security/pict/enableAuditing.png b/content/security/pict/enableAuditing.png index 215cd597a3..32a0fa6fc6 100644 Binary files a/content/security/pict/enableAuditing.png and b/content/security/pict/enableAuditing.png differ diff --git a/content/security/pict/eventFilteringToggle.png b/content/security/pict/eventFilteringToggle.png new file mode 100644 index 0000000000..7b6a27d3e5 Binary files /dev/null and b/content/security/pict/eventFilteringToggle.png differ diff --git a/content/security/pict/eventFilteringUIdataServiceEnabled.png b/content/security/pict/eventFilteringUIdataServiceEnabled.png new file mode 100644 index 0000000000..93f51601e5 Binary files /dev/null and b/content/security/pict/eventFilteringUIdataServiceEnabled.png differ diff --git a/content/security/pict/eventFilteringUIdataServiceInitial.png b/content/security/pict/eventFilteringUIdataServiceInitial.png new file mode 100644 index 0000000000..93f4d68879 Binary files /dev/null and b/content/security/pict/eventFilteringUIdataServiceInitial.png differ diff --git a/content/security/pict/manageUserNew.png b/content/security/pict/manageUserNew.png index 2cc9dc921a..f3aa2e308e 100644 Binary files a/content/security/pict/manageUserNew.png and b/content/security/pict/manageUserNew.png differ diff --git a/content/security/pict/manageUserNewSubsequent2.png b/content/security/pict/manageUserNewSubsequent2.png index 9890f0b2f7..e13096f400 100644 Binary files a/content/security/pict/manageUserNewSubsequent2.png and b/content/security/pict/manageUserNewSubsequent2.png differ diff --git a/content/security/pict/n1ql-auditing_1_open.png b/content/security/pict/n1ql-auditing_1_open.png new file mode 100644 index 0000000000..fbd4ac5bbf Binary files /dev/null and b/content/security/pict/n1ql-auditing_1_open.png differ diff --git a/content/security/pict/n1ql-auditing_2_n1ql-events.png b/content/security/pict/n1ql-auditing_2_n1ql-events.png new file mode 100644 index 0000000000..714fb3eb1a Binary files /dev/null and b/content/security/pict/n1ql-auditing_2_n1ql-events.png differ diff --git a/content/security/pict/requireClientCertificatePanel.png b/content/security/pict/requireClientCertificatePanel.png new file mode 100644 index 0000000000..7bc773b36d Binary files /dev/null and b/content/security/pict/requireClientCertificatePanel.png differ diff --git a/content/security/pict/rolesPanelAdminChecked.png b/content/security/pict/rolesPanelAdminChecked.png index 928752b441..7d8c59a72b 100644 Binary files a/content/security/pict/rolesPanelAdminChecked.png and b/content/security/pict/rolesPanelAdminChecked.png differ diff --git a/content/security/pict/securityView.png b/content/security/pict/securityView.png index b1d7c737ba..30ea74ba11 100644 Binary files a/content/security/pict/securityView.png and b/content/security/pict/securityView.png differ diff --git a/content/security/pict/securityViewInitial.png b/content/security/pict/securityViewInitial.png index 5ae595bdc2..50433971fe 100644 Binary files a/content/security/pict/securityViewInitial.png and b/content/security/pict/securityViewInitial.png differ diff --git a/content/security/pict/securityViewInitialNoUsers.png b/content/security/pict/securityViewInitialNoUsers.png index 1173143bf0..e778797da0 100644 Binary files a/content/security/pict/securityViewInitialNoUsers.png and b/content/security/pict/securityViewInitialNoUsers.png differ diff --git a/content/security/pict/securityViewWithNewUser.png b/content/security/pict/securityViewWithNewUser.png index d4dc35542d..427a8162b4 100644 Binary files a/content/security/pict/securityViewWithNewUser.png and b/content/security/pict/securityViewWithNewUser.png differ diff --git a/content/security/pict/sessionScreen.png b/content/security/pict/sessionScreen.png new file mode 100644 index 0000000000..5fa7b37bea Binary files /dev/null and b/content/security/pict/sessionScreen.png differ diff --git a/content/security/pict/sessionTimeOutTab.png b/content/security/pict/sessionTimeOutTab.png new file mode 100644 index 0000000000..c8dc7a1c85 Binary files /dev/null and b/content/security/pict/sessionTimeOutTab.png differ diff --git a/content/security/pict/setRotationTimeInterval.png b/content/security/pict/setRotationTimeInterval.png index 1edc7bdab6..56bfa07fe1 100644 Binary files a/content/security/pict/setRotationTimeInterval.png and b/content/security/pict/setRotationTimeInterval.png differ diff --git a/content/security/secret-mgmt.dita b/content/security/secret-mgmt.dita index a3fdfd4178..34a6f3a098 100644 --- a/content/security/secret-mgmt.dita +++ b/content/security/secret-mgmt.dita @@ -45,7 +45,7 @@

    Secret-Management is an optional feature that only works when the master password is set for each Couchbase Server node. This can be specified by means of the - master-password CLI command; + master-password CLI command; the REST API POST /node/controller/changeMasterPassword method; and (on all supported operating systems other than Mac OS X) by explicitly @@ -120,7 +120,7 @@

  • Master-password rotation: This first level of rotation is achieved by setting a new password, using the - CLI command, the + CLI command, the REST API method, or (other than on Mac OS X) the environment variable, as indicated above. One master password per node needs to be set. @@ -133,7 +133,7 @@
  • Data-key rotation: This second level of rotation is achieved by changing the data-key, - using the CLI command with + using the CLI command with the --rotate-data-key option, or the REST API POST /node/controller/rotateDataKey method. @@ -146,7 +146,7 @@
  • Secret rotation: This third level of rotation is achieved by changing the values of the secrets themselves. For example, to reset the secret that is an administrator password, use the - couchbase-cli reset-admin-password + couchbase-cli reset-admin-password command.

    diff --git a/content/security/security-admin-access.dita b/content/security/security-admin-access.dita index 5f9d962446..9aa0630714 100644 --- a/content/security/security-admin-access.dita +++ b/content/security/security-admin-access.dita @@ -41,7 +41,7 @@

  • The Root Certificate link shows that a self-signed certificate has been deployed across the cluster. This certificate can be regenerated or replaced using or CLI commands for managing + href="../cli/cbcli/couchbase-cli-ssl-manage.html" format="html">CLI commands for managing certificates.
    • diff --git a/content/security/security-admins.dita b/content/security/security-admins.dita index 9af1bcf551..605b5b927c 100644 --- a/content/security/security-admins.dita +++ b/content/security/security-admins.dita @@ -39,7 +39,7 @@
    Resetting the administrative password The administrative password can be reset using the password reset tool: + href="../cli/cbcli/couchbase-cli-reset-admin-password.html" format="html"/>
    diff --git a/content/security/security-audit-events.dita b/content/security/security-audit-events.dita index 52dea3ca6d..ced8144a1f 100644 --- a/content/security/security-audit-events.dita +++ b/content/security/security-audit-events.dita @@ -133,6 +133,11 @@ Compaction settings modified + + Bucket compression mode modified + Bucket TTL modified + +
    diff --git a/content/security/security-audit-targets.dita b/content/security/security-audit-targets.dita index d40fe2d6c0..65ab8c9ed1 100644 --- a/content/security/security-audit-targets.dita +++ b/content/security/security-audit-targets.dita @@ -20,12 +20,14 @@

    - Auditable events are captured in JSON files. A default file, named audit.log is - always available; and logged events are written to this file. After an administrator-specified - period — which must be a minimum of 15 minutes and a maximum of 7 days — the default - file is saved under a modified name that features a timestamp corresponding to the time of saving. - A new, empty audit.log file is saved, and logged events continue to be written to - this new file. (Note that rollover may happen earlier if the file reaches its maximum size of 20MB.) + When auditing is enabled, logged events are written to a default file, + named audit.log. After an administrator-specified + period — which must be a minimum of 15 minutes and a maximum of 7 days — this + file is closed, and is saved under a modified name that features a timestamp corresponding + to the time of saving. + A new, empty audit.log file is + created and saved when a new audit event is generated. Note that this rotation may happen earlier if the file + reaches its maximum size of 20MB. For instructions on configuring the file's rotation time, see Auditing.

    @@ -43,15 +45,17 @@ { "timestamp":"2015-02-20T08:48:49.408-08:00", "id":8192, - "name":"login success", - "description":"Successful login to couchbase cluster", - "role":"admin", - "real_userid": { - "source":"ns_server", - "user":"bjones" - }, - "sessionid":"0fd0b5305d1561ca2b10f9d795819b2e", - "remote":{"ip":"172.23.107.165", "port":59383} + "name":"login success", + "description":"Successful login to couchbase cluster", + "role":"admin", + "real_userid": { + "source":"ns_server", + "user":"bjones" + }, + "sessionid":"0fd0b5305d1561ca2b10f9d795819b2e", + "remote":{ + "ip":"172.23.107.165", "port":59383 + } }

    @@ -72,20 +76,19 @@

    { - "real_userid": { - "source": "rejected", - "user": "auditBucketUser" - }, - "remote": { - "ip": "127.0.0.1", - "port": 64416 - }, - "timestamp": "2017-03-16T15:45:27.420Z", - "id": 8193, - "name": "login failure", - "description": "Unsuccessful attempt to login to couchbase cluster" -} - + "real_userid": { + "source": "rejected", + "user": "auditBucketUser" + }, + "remote": { + "ip": "127.0.0.1", + "port": 64416 + }, + "timestamp": "2017-03-16T15:45:27.420Z", + "id": 8193, + "name": "login failure", + "description": "Unsuccessful attempt to login to couchbase cluster" +}

    This record indicates that a user named auditBucketUser incurred an @@ -104,45 +107,83 @@

    { - "props": { - "storage_mode": "couchstore", - "conflict_resolution_type": "seqno", - "eviction_policy": "value_only", - "num_threads": 3, - "flush_enabled": false, - "purge_interval": "undefined", - "auth_type": "sasl", - "ram_quota": 1156579328, - "replica_index": false, - "num_replicas": 1 - }, - "type": "membase", - "bucket_name": "auditBucket", - "real_userid": { - "source": "ns_server", - "user": "Administrator" - }, - "sessionid": "dca284b5efe1937a1a4085ef88c2fbcb", - "remote": { - "ip": "127.0.0.1", - "port": 64477 - }, - "timestamp": "2017-03-16T15:43:35.187Z", - "id": 8201, - "name": "create bucket", - "description": "Bucket was created" -} - + "props":{ + "compression_mode":"off", + "max_ttl":12000, + "storage_mode":"couchstore", + "conflict_resolution_type":"seqno", + "eviction_policy":"value_only", + "num_threads":3, + "flush_enabled":false, + "purge_interval":"undefined", + "ram_quota":163577856, + "replica_index":false, + "num_replicas":1 + }, + "type":"membase", + "bucket_name":"ProductionBucket", + "real_userid":{ + "source":"ns_server", + "user":"Administrator" + }, + "sessionid":"5dd53fe63703c7fdc45ff75596e39a35", + "remote":{ + "ip":"127.0.0.1", + "port":61908 + }, + "timestamp":"2018-02-07T15:22:54.960Z", + "id":8201, + "name":"create bucket", + "description":"Bucket was created" +}

    - This record indicates that a Bucket was created on 2017-03-16 at - 15:43:35; that the bucket was named auditBucket; that it was created - with sasl authentication-access required, and that its eviction-policy was defined + This record indicates that a Bucket was created on 2018-02-07 at + 15:22:54; that the bucket was named ProductionBucket; and that its eviction-policy was defined as value_only. The bucket was created by the system's full Administrator.

    +
    + + Bucket TTL Modification + + +

    + The audit-record below corresponds to the modification of Bucket TTL, for the bucket + created immediately above. +

    + + { + "props":{ + "max_ttl":15000, + "storage_mode":"couchstore", + "eviction_policy":"value_only", + "num_threads":3, + "flush_enabled":false, + "purge_interval":"undefined", + "ram_quota":163577856, + "num_replicas":1 + }, + "type":"membase", + "bucket_name":"ProductionBucket", + "real_userid":{ + "source":"ns_server", + "user":"Administrator" + }, + "sessionid":"12774a2e146c650eeed8c6d9486857ad", + "remote":{ + "ip":"127.0.0.1","port":61966 + }, + "timestamp":"2018-02-07T15:23:51.350Z", + "id":8202, + "name":"modify bucket", + "description":"Bucket was modified" +} + +
    +
    User Creation @@ -153,26 +194,26 @@ </p> <codeblock outputclass="language-json">{ - "roles": [ - "ro_admin" - ], - "identity": { - "source": "builtin", - "user": "auditBucketUser2" - }, - "real_userid": { - "source": "ns_server", - "user": "Administrator" - }, - "sessionid": "dca284b5efe1937a1a4085ef88c2fbcb", - "remote": { - "ip": "127.0.0.1", - "port": 64416 - }, - "timestamp": "2017-03-16T15:44:32.254Z", - "id": 8232, - "name": "set user", - "description": "User was added or updated" + "roles": [ + "ro_admin" + ], + "identity": { + "source": "builtin", + "user": "auditBucketUser2" + }, + "real_userid": { + "source": "ns_server", + "user": "Administrator" + }, + "sessionid": "dca284b5efe1937a1a4085ef88c2fbcb", + "remote": { + "ip": "127.0.0.1", + "port": 64416 + }, + "timestamp": "2017-03-16T15:44:32.254Z", + "id": 8232, + "name": "set user", + "description": "User was added or updated" } </codeblock> @@ -194,15 +235,15 @@ </p> <codeblock outputclass="language-json">{ - "timestamp": "2017-03-16T16:12:36.198Z", - "real_userid": { - "source": "ns_server", - "user": "Administrator" - }, - "index_name": "def-airportname", - "id": 24577, - "name": "Create/Update index", - "description": "FTS index was created/Updated" + "timestamp": "2017-03-16T16:12:36.198Z", + "real_userid": { + "source": "ns_server", + "user": "Administrator" + }, + "index_name": "def-airportname", + "id": 24577, + "name": "Create/Update index", + "description": "FTS index was created/Updated" }</codeblock> <p> @@ -211,8 +252,6 @@ </p> </section> - - - + </conbody> </concept> diff --git a/content/security/security-auditing.dita b/content/security/security-auditing.dita index 750e4e64ea..fda0228384 100644 --- a/content/security/security-auditing.dita +++ b/content/security/security-auditing.dita @@ -6,68 +6,23 @@ - Actions performed on Couchbase Server can be audited. This allows a full - administrator to ensure that system-management tasks are being appropriately - performed. This potentially facilitates compliance with regulatory standards. + Actions performed on Couchbase Server can be audited. This allows administrators + to ensure that system-management tasks are being appropriately + performed.
    - Audit Records and their Content + Audit Records and Their Content

    The records created by the Couchbase Auditing facililty capture information on - who has performed what action, when, and how successfully: -

    - -
      -
    • - Who: The administrator performing actions. To access the system, each - administrator has authenticated, either locally or by means of LDAP: and is therefore - identifiable throughout their session. - -

      - -

      - -
      • - -
    • - What: The action performed. 30 different kinds of action are tracked by Couchbase Server. - -

      - -

      - -
      • - -
    • - When: The UTC time stamp that corresponds to each recorded action. - -

      - -

      - -
      • - -
    • - How: The success or failure of the action. - -

      - -

      - -
      • - -
    - -

    - Audit records are created by Couchbase Server-processes, which run asynchronously. Each - record is stored as a JSON file, - which can be retrieved and inspected. + who has performed what action, when, and how successfully. + The records are created by Couchbase Server-processes, which run asynchronously. Each + record is stored as a JSON document, which can be retrieved and inspected.

    @@ -79,12 +34,13 @@

    - Only a Full Administrator can configure auditing. Configuration is performed by - means of the Couchbase Web Console. Proceed as follows. + Auditing can be configured by the Full Admin and Security Admin roles, + and read by the Read-Only Admin role, using Couchbase Web Console. + Proceed as follows.

    - Access the Couchbase Web Console, and left-click on the Security tab, + Access Couchbase Web Console, and left-click on the Security tab, in the vertical navigation-bar, at the left-hand side of the Dashboard:

    @@ -118,57 +74,130 @@

    - To enable auditing, check the Enable Auditing checkbox: + To enable auditing, check the Audit events & write them to a log checkbox:

    - +

    - This makes the default pathname within the Target Log Directory text-field - editable. If you wish to modify the pathname, enter the appropriate content. Records will be saved - to the directory you specify. + This makes the default pathname within the Audit Log Directory text-field + editable. For Linux, the pathname is /opt/couchbase/var/lib/couchbase/logs; for + Windows, C:\Program Files\Couchbase\Server\var\lib\couchbase\logs; for MacOS, + /Users/couchbase/Library/Application Support/Couchbase/var/lib/couchbase/logs. +

    + + +

    + If you wish to modify the pathname, enter the appropriate content. Records will be saved + to the directory you specify. Note the advisory message now visible beneath the checkbox: as this + indicates, electing to audit a wide range of events may significantly impact performance + and consume disk-space.

    + +

    - The Log Rotation Time Interval determines how often stored log files — referred + The Log Rotation time interval & size trigger determines at + what times stored log files — referred to as targets — are rotated: this means that the current default file, to which records are being written, named - audit.log is + audit.log, is saved under a new name, which features an appended timestamp. For example: usermachinename.local-2017-03-16T15-42-18-audit.log.

    - The number of units is specified + The number of time-units is specified by changing the number 1, which appears in the interactive field by default. The - unit-type is specified by means of the pull-down menu, at the right-hand side of the field: + time-unit type is specified by means of the pull-down menu, at the right-hand side of the field: +

    + +

    +

    - + Note that the value you establish must be from 15 minutes to 7 days.

    - Note that the value you establish must be in the range from 15 minutes to 7 days. + Log rotation can also be specified by means of a size trigger: this can be edited, in the + interactive field to the right of the Log Rotation pane. The default value is 20, and the + units are megabytes.

    -
    Configuring with CLI -

    The following CLI syntax is used to configure Couchbase auditing for administrators:

    +
    + + + Filterable Events + + +

    + Events can be filtered for the Data Service, the Query and Index + Service, and the Eventing Service. Filtering means selective + logging. +

    - couchbase-cli - setting-audit OPTIONS - --audit-log-rotate-interval=[MINUTES] //log rotation interval - --audit-log-path=[PATH] //target log directory - --audit-enabled=[0|1] //enable auditing or not +

    + To view filterable events for the Data Service, first, ensure that logging is generally enabled, + by checking the + Audit events & write them to a log checkbox. Then, left-click + on the right-pointing arrowhead adjacent to Data Service. The Data Service + events panel opens, as follows: +

    + +

    + +

    + +

    + This shows that currently, no Data Service events are to be logged. To elect to log all such + events, move the enable all toggle to the right: +

    + +

    + +

    + +

    + The panel now appears as follows: +

    + +

    + +

    + +

    + Every checkbox appears selected, indicating that each corresponding event will be logged. To de-select + individual events, simply uncheck the appropriate checkboxes. +

    + +

    + In some cases, it may not be desirable to log events incurred by particular users: for example, authentication + performed by the Full Administrator. These users can be specified in the Ignore Filterable Events From These + Users field. As the placeholder indicates, specification should take the form + username/external or username/couchbase, according to + the domain in which the user is registered. (See + Authentication, for + information on authentication domains.) + Left-click on the Save button, to save the list of users. +

    + + +
    + +
    + + Configuring with CLI +

    - See - setting-audit - for details. + For information on configuring audit with the Couchbase command-line interface, see + setting-audit.

    @@ -189,6 +218,6 @@ - + diff --git a/content/security/security-authentication.dita b/content/security/security-authentication.dita index 4aecf54c6e..58cfad4317 100644 --- a/content/security/security-authentication.dita +++ b/content/security/security-authentication.dita @@ -80,12 +80,13 @@
  • - Generated Users, which are created by Couchbase Server as part of the upgrade process to - 5.0; each in correspondence with a legacy bucket. Each Generated User is assigned a username + Generated Users, which are created by Couchbase Server as part of the upgrade process from pre-5.0 + to 5.0 and post-5.0 versions; + each in correspondence with a legacy bucket. Each Generated User is assigned a username that is identical to the bucket-name; and either a password that is identical to the bucket's pre-5.0 password, or no password, if the bucket did not feature a password. Generated Users are created to ensure that legacy applications can continue to access legacy buckets after upgrade to - 5.0, with the same username-password combination being used for authentication. + 5.0 or post-5.0, with the same username-password combination being used for authentication.

    diff --git a/content/security/security-authorization.dita b/content/security/security-authorization.dita index f76ee80fe5..ae3df7b9c1 100644 --- a/content/security/security-authorization.dita +++ b/content/security/security-authorization.dita @@ -29,7 +29,7 @@ Couchbase Server Enterprise Edition provides RBAC with multiple roles for finer access control. Community Edition provides multiple users that can be assigned to limited set of roles. There are three fixed roles in the community edition of Couchbase providing - coarser access control. + coarser access control: Bucket Full Access (bucket_full_access[*]), Admin (admin), and Read Only Admin (ro_admin).

    • @@ -132,10 +132,7 @@ Privilege: The right, assigned by Couchbase Server, to apply an action to a resource. Possible actions include read, write, and execute. The associations of privileges to resources and roles are explained in - RBAC for - Administrators and - RBAC for - Applications. + Roles.

    @@ -146,10 +143,7 @@

  • Role: An entity associated with a fixed set of privileges. The association of privileges to roles is described in - RBAC for - Administrators and - RBAC for - Applications. + Roles.

    diff --git a/content/security/security-certs-auth.dita b/content/security/security-certs-auth.dita index cae8f1a759..6f99e10f17 100644 --- a/content/security/security-certs-auth.dita +++ b/content/security/security-certs-auth.dita @@ -6,8 +6,8 @@

    With the growing threat to security from rogue users and/or machines, password-based authentication is no longer a reliable method of authenticating users. Couchbase now supports the use of x.509 certificates to authenticate clients which ensures that only approved users - (or machines or endpoints) are authenticated. Note that this authentication mechanism is - available only for the data service.

    + (or machines or endpoints) are authenticated. This authentication mechanism can be used by + the Couchbase SDK to access the Data, Query, and Search Services.

    Certificate based authentication provides an additional layer of security. It relies on a certificate authority, CA, to validate identities and issue certificates. The certificate includes information such as the name of the entity it identifies, an expiration date, the @@ -26,12 +26,47 @@

  • When the client presents the certificate, the server checks for the validity of the certificate.
  • If the certificate is valid and not expired, then it parses the certificate to obtain - the user specified in the certificate.
    • + the user name specified in the certificate.
  • The server then verifies the user and the roles assigned to the user. If the user has appropriate roles, then it authenticates the user and completes the handshake.
  • If any of the checks fail, the handshake is not completed.

    • +

    Since the identity in client certificates can be encoded in different ways for different + kinds of users, Couchbase provides a flexible scheme to extract the identity from different + certificates. Starting version 5.1, Couchbase Server supports multiple prefixes to be + specified for client certificate authentication, as shown below.

    +

    + "client_cert_auth": { + "state": "enable", + "prefixes": [ + { + "path": "san.dnsname", + "prefix": "www.", + "delimiters": ".,;" + }, + {}, + ... + ] +} +

    +

    The "prefixes" field is an array of {path, prefix, + delimiters} tuples, where:

      +
    • path can be one of the following values: + "subject.cn", "san.uri", "san.email", + "san.dns"
      • +
    • prefix specifies the client certificate prefix value.
      • +
    • delimiters is a list of characters that can individually be + treated as a delimiter
      • +

    + +

    The following constraints apply when specifying multiple tuples:

      +
    • You can specify up to a maximum of 10 {path, prefix,delimiters} tuples + in the "prefixes" array.
      • +
    • No two tuples can have the same "path" and + "prefix" fields.
      • +
    • All the fields in the tuple must be specified.
      • +

    Prerequisites

    @@ -59,19 +94,32 @@

    Using REST API

      -
    1. Enable x.509 certificate authentication on the server. You must set this parameter - once per - cluster.curl -X POST --data-binary "state=enable" http://Administrator:password@localhost:8091/settings/clientCertAuth
      2. -
    2. Specify the path of the certificate to use to parse for the user on the server. - curl -X POST --data-binary "path=subject.cn" http://Administrator:password@localhost:8091/settings/clientCertAuth
      3. -
    3. Specify the prefix and delimiter in that path expression. - curl -X POST --data-binary "delimiter=" http://Administrator:password@localhost:8091/settings/clientCertAuth -curl -X POST --data-binary "prefix=" http://Administrator:password@localhost:8091/settings/clientCertAuth
      4. +
    4. Create a data file containing the payload for your certificate authentication + settings. For example, cb-certauth-setting.json: + { + "state": "enable", + "prefixes": [{ + "path": "san.uri", + "prefix": "www.", + "delimiter": "." + }, + { + "path": "san.email", + "prefix": "couchbase.", + "delimiter": "." + } + ] +}
      5. +
    5. Run the following command to enable x.509 certificate authentication on the + server.curl -H "Content-Type: application/json" --data-binary @cb-certauth-setting.json http://Administrator:password@127.0.0.1:8091/settings/clientCertAuth

    +

    To retrieve the client certificate authentication settings, run the following command: + curl -X GET Administrator:password@localhost:8091/settings/clientCertAuth{"state":"enable","prefixes":[{"delimiter":".","path":"san.uri","prefix":"www."},{"delimiter":".","path":"san.email","prefix":"couchbase."}]}

    Using CLI

    -

    Use the following couchbase-cli command: - couchbase-cli ssl-manage <options>

    +

    Use the couchbase-cli + + command: couchbase-cli ssl-manage <options>

    @@ -104,8 +152,10 @@ curl -X POST --data-binary "prefix=" http://Administrator:password@localhost:80 --set-client-auth-path - subject.cn | san.uri | san.dnsname | san.name - Set SSL client certificate type value. + subject.cn | san.uri | san.dnsname | san.email + Set SSL client certificate type value. This field will be used to extract the + user name from the certificate. Currently, only the fields specified in the values + column are supported. --set-client-auth-prefix @@ -115,7 +165,9 @@ curl -X POST --data-binary "prefix=" http://Administrator:password@localhost:80 --set-client-auth-delimiter set_client_auth_delimiter - Set SSL client certificate delimiter value. + Set SSL client certificate delimiter value. The delimiter can either be a + string or a character. The parsing of the certificate for the user name ends when + the delimiter value is found. --client-auth @@ -138,10 +190,19 @@ curl -X POST --data-binary "prefix=" http://Administrator:password@localhost:80
    Limitations

    Note the following limitations to the feature in the current release:

      -
    • X.509 Certificate Based Authentication will only work for data service.
      • -
    • For Couchbase Server 5.0, X.509 Certificate-based Authentication support is only available in the Java Client.
      • -
    -

    +
  • X.509 Certificate Based Authentication will only work for data service.
    • +
  • For Couchbase Server 5.5, X.509 Certificate-based Authentication is suported + by all SDK Clients. However, only the very latest versions support it - check the release notes for your SDK version.
    • +

    +
    +
    Upgrade +

    When upgrading from an earlier version to 5.5, the cluster will be in mixed mode and will + return client certificate authentication settings in the older format until the cluster is + completely upgraded. Once the cluster has been upgraded, any existing client certificate + authentication settings from earlier versions will be automatically transformed into the new + format.

    diff --git a/content/security/security-comm-encryption.dita b/content/security/security-comm-encryption.dita index e778f2dc1f..7c0184cc45 100644 --- a/content/security/security-comm-encryption.dita +++ b/content/security/security-comm-encryption.dita @@ -89,42 +89,87 @@

    -

    +

    Note that the certificate can be regenerated or replaced, using the , or - CLI commands + CLI commands for certificate-management.

    + +
    + + + Handling Client Certificates + + +

    + Couchbase Server supports authentication based on client-side certificates. An example of client-side certificate-generation is provided in + Provide Certificate-Based Authentication + for a Java Client; and an example of how such a certificate can be referenced by a client-program is provided in + Authenticating + a Java Client by Certificate. +

    + +

    + Clients' use of certificates can be enabled, disabled, or made mandatory for the cluster; by + means of the Client Certificate screen, which is accessible from the Client Certificate tab, near the top of + the main Security screen of Couchbase Web Console. +

    + +

    + +

    + +

    + Select Disable to disable certificate-based authentication: this means that authorization can only occur by means + of username and password (for Couchbase-defined users) or username (for LDAP-defined users). Select Enable to make + certificate-based authentication optional for clients. Select Mandatory to make certificate-based authentication a + requirement for clients (except in the case of manual login, at the Couchbase Web Console). +

    + +

    + Appropriate paths, prefixes, and delimiters + can be specified: for detailed information on certificate-identity extraction, see + Certificate-Based Authentication. +

    + +
    - Disabling Non-Secure Console-Access + Disabling Non-Secure Console Access

    If you wish to force Administrators to log into the UI over an encrypted channel, and so use port 18091, - you can disable the Couchbase Web Console on port 8091. This only affects console-access, and + you can disable Couchbase Web Console access on port 8091. This only affects console access, and does not prevent REST requests from continuing to use 8091, non-securely.

    -

    - The following REST API method disables the Couchbase Web Console on port 8091: - curl -X POST -u Administrator:password http://127.0.0.1:8091/diag/eval \ - -d 'ns_config:set(disable_ui_over_http, true)' -

    +

    The following REST API method disables the Couchbase Web Console on port 8091: + curl -u Administrator:password http://localhost:8091/settings/security -d disableUIOverHttp=true +

    - The following REST API method re-enables the Couchbase Web Console on port 8091: - curl -X POST -u Administrator:password http://127.0.0.1:8091/diag/eval \ - -d 'ns_config:set(disable_ui_over_http, false)' + To re-enable Couchbase Web Console access on port 8091: + curl -u Administrator:password http://localhost:8091/settings/security -d disableUIOverHttp=false

    - + + +

    + The command line can also be used for this disabling step: +

    + + couchbase-cli setting-security -c <servername> -u Administrator \ +   -p password --disable_ui_over_http + +
    @@ -161,7 +206,8 @@ Encryption for data access is performed through client-server communication and view access.

    - + **This one line** Added 2018/01/12 - let's remove this Question & commented out section. Objections? + -->
    @@ -214,56 +260,114 @@

    Port 18092 must be used for secure view access. Thus: - https://servername:18092 + https://servername:18092.

    + +
    + + + Securing Query Access + + +

    + The + REST endpoint for secure access to N1QL queries is: + https://servername:18093/query/service. +

    + +
    + +
    + + Securing FTS Access + + +

    + To use Full Text Search securely, port 18094 must be specified. For + example: +

    + + curl -u Administrator:password https://localhost:18094/api/index/myFirstIndex/count + +
    + +
    + + + Securing Backups + +

    + Both + cbbackup and + cbbackupmgr can use the secure port: https://servername:18091. +

    +
    +
    Overriding Supported Ciphers +

    + Couchbase Server uses ciphers that are accepted by default by OpenSSL.The default behavior + employs high-security ciphers, built into OpenSSL. +

    + + +

    + You can override this selection, by setting the COUCHBASE_SSL_CIPHER_LIST environment + variable — this must be performed before starting Couchbase Server. The environment + variable can be set in either of the following ways: +

    + +
      + +

    • - Couchbase Server uses ciphers that are accepted by default by OpenSSL.The default behavior - employs high-security ciphers, built into openSSL. On macOS these are: + Specify an explicit list of ciphers to be used. For example:

      - -
        -
      • AES256-SHA
        • -
      • AES128-SHA
        • -
      • DES-CBC3-SHA
        • -
      - + + COUCHBASE_SSL_CIPHER_LIST="DHE-DSS-AES128-SHA,CAMELLIA128-SHA" +

      - You can override this selection, by setting the following environment variable before - starting Couchbase: +

      - - COUCHBASE_SSL_CIPHER_LIST= <list-of-ciphers-to-accept> - +
      • + +

    • - If you wish to include both medium and high-security ciphers in your installation, you - can define the environment variable as follows: + Specify ciphers by security-level. For example, to specify that all ciphers in both + medium and high categories be used, enter the following:

      - - COUCHBASE_SSL_CIPHER_LIST= MEDIUM, HIGH - + + COUCHBASE_SSL_CIPHER_LIST="MEDIUM,HIGH" +

      - For macOS, the ciphers so specified would be the following -

      - -
        -
      • SEED-SHA
        • -
      • AES256-SHA
        • -
      • AES128-SHA
        • -
      • DES-CBC3-SHA
        • -
      • RC4-SHA
        • -
      • RC4-MD5
        • -
      + +

      +
      • + +
    + +

    + To display the ciphers available on your Linux platform for a particular security level, use + the openssl command. For example, to display the high-level ciphers, + enter the following: +

    + openssl ciphers -v 'HIGH' + +

    + To check the current value of the COUCHBASE_SSL_CIPHER_LIST environment variable, + type printenv at the Linux prompt: this returns a list of all currently set + environment variables. +

    +
    diff --git a/content/security/security-ldap-gui-new.dita b/content/security/security-ldap-gui-new.dita index 49cdc67b30..55cfc78302 100644 --- a/content/security/security-ldap-gui-new.dita +++ b/content/security/security-ldap-gui-new.dita @@ -70,7 +70,7 @@ Full administrators in Couchbase can manage certificates using the Couchbase - CLI tools (as described in ) or REST + CLI tools (as described in ) or REST API (as described in ). @@ -87,6 +87,6 @@
    - + diff --git a/content/security/security-ldap-tbls-new.dita b/content/security/security-ldap-tbls-new.dita index 76f84a06af..6924034c65 100644 --- a/content/security/security-ldap-tbls-new.dita +++ b/content/security/security-ldap-tbls-new.dita @@ -21,7 +21,7 @@
    Errors caused by saslauthd:
    ..... - "dial unix /var/run.sasl2/mux: no such file or directory" + "dial unix /var/run/sasl2/mux: no such file or directory"

    Possible reason for the error: no LDAP server was installed.

    • Before installing the server again, you can try to resolve the issue by creating a diff --git a/content/security/security-pam-auth.dita b/content/security/security-pam-auth.dita index a32b629071..4364dedb7f 100644 --- a/content/security/security-pam-auth.dita +++ b/content/security/security-pam-auth.dita @@ -78,6 +78,13 @@ when administrators log into Couchbase Server. Supervisor access, via sudo, is required to perform most of the steps; and an editor is required, to allow you to edit configuration files. +

      + + For PAM to be fully configured, the following procedure must be performed on each + node in the cluster. + + +

      Proceed as follows:

      @@ -109,7 +116,7 @@

      Note that --ldap-enabled 1 enables external authentication, and --ldap-enabled 0 disables. See - setting-ldap for further information. When + setting-ldap for further information. When successfully executed, the command provides the following notification: SUCCESS: LDAP settings modified.

      • diff --git a/content/security/security-passwords.dita b/content/security/security-passwords.dita index aec7011d71..7a59328f7b 100644 --- a/content/security/security-passwords.dita +++ b/content/security/security-passwords.dita @@ -72,7 +72,7 @@ password-policy. For production purposes, it should be considered essential to do so. The password-policy can be set by the Full Administrator, by means of the Couchbase CLI command - setting-password-policy. This + setting-password-policy. This allows you to specify that every Couchbase Server-password should:

      @@ -114,7 +114,7 @@

      You can reset any forgotten administrative passwords using the Couchbase CLI command reset-admin-password. reset-admin-password. We strongly suggest that you use strong user passwords, and if you want to enforce a strong password, you should use a strong password policy. Password enforcement is done during password setting and rotation.

      @@ -145,7 +145,10 @@ To authenticate, every Couchbase Server-user must specify a username as well as a password. The restrictions on username-design are that each should be unique to the cluster; and that none of the following characters be - used: ( ) < > @ , ; : \ " / [ ] ? = { }. + used: ( ) < > @ , ; : \ " / [ ] ? = { }. + Usernames may not be more than 128 UTF-8 characters in length; and + it is recommended that they be no more than 64 UTF-8 characters in length, in order to ensure + successful onscreen display.

      diff --git a/content/security/security-pw-auth.dita b/content/security/security-pw-auth.dita index cfdbb3043c..77afb9d80d 100644 --- a/content/security/security-pw-auth.dita +++ b/content/security/security-pw-auth.dita @@ -28,13 +28,12 @@ Initialize the Cluster for details. Subsequently, passwords can be changed by means of the password-reset tool, - +

      - Best practices for password-definition are - listed in the section - Couchbase passwords. + For best practices for password-definition, and restrictions on username-design, see + Couchbase Passwords.

      diff --git a/content/security/security-rbac-for-admins-and-apps.dita b/content/security/security-rbac-for-admins-and-apps.dita index abc3a44d7c..0bc53c46b1 100644 --- a/content/security/security-rbac-for-admins-and-apps.dita +++ b/content/security/security-rbac-for-admins-and-apps.dita @@ -52,13 +52,15 @@

      The Security view allows users to be defined, and roles to be allocated to them. It also - allows management of the Root Certificate, and of Audit-processing. + allows management of the Root and Client certificates; of + Audit-processing; of Log Redaction; and of Session + timeouts. To add a user, left-click on the Add User control, at the upper right. The Add New User dialog now appears:

      - +

      @@ -92,59 +94,60 @@

      - To specify one or more roles, scroll down, and review the + Roles can be specified by checking checkboxes in the Roles panel:

      - +

      - Each available role can be selected by means of a checkbox. Roles are arranged in a hierarchy: by left-clicking - on right-pointing arrowheads, you open a lower level of this hierarchy, and are able to inspect either - additional roles, or additional levels, or both. For example, left-click successively on - the arrowheads for Data Roles, Data Reader, - and Data Writer: + Roles are arranged in groups. The + first group, which appears at the top of the panel, is for Administration & Global Roles: this + roles are either administrative, or involve access to cluster-wide features. Subsequent + groups appear below the first: each consists of roles that are applied per bucket. + By default, a single subsequent group is displayed, All Buckets (*). Each individual bucket + on the cluster will be represented below All Buckets (*), with its + own hierarchy of roles. This allows each individual user to be assigned roles either that apply to all + buckets defined on the cluster, or to individual buckets. + Left-click on the corresponding right-pointing arrowhead to display a group's contents, scrolling down + if necessary:

      - +

      - When opened, Data Reader, - and Data Writer each - reveal two checkboxes; which assign read and write permission respectively - to all buckets, and to - the travel-sample bucket alone. - To assign the user a specific role, check each appropriate checkbox, by left-clicking. For example: + To display roles at lower levels of the All Buckets (*) hierarchy, left-click on the right-pointing + arrowheads.

      - +

      - These role-assignments give the user read and write permission on the data in the - travel-sample bucket. + To assign roles to the user, simply check the appropriate checkboxes. Then, left-click on the Add User + button, at the lower right.

      Note that some roles are considered to be subsets of others. In such cases, manually checking one checkbox may trigger the automated checking of others — indicating that the corresponding roles are also assigned to the user. - To demonstrate this, left-click on the Admin checkbox. The Roles + To demonstrate this, left-click on the Full Admin checkbox, near the top. The Roles panel now appears as follows:

      - +

      - As illustrated, selecting the Admin role causes all other roles - also to become selected: this is because Admin stands at the top of the + As illustrated, selecting the Full Admin role causes all other roles + also to become selected: this is because Full Admin stands at the top of the hierarchy, and is a superset of all other roles.

      @@ -157,15 +160,16 @@

      - Whenever you have finished allocating roles to a particular user, left-click on Save. + Whenever you have finished allocating roles to a particular user, left-click on Add User. The dialog disappears, and the Security view now displays, on the row of the corresponding username, the roles you have allocated. For example, if you - have allocated Data Reader and Data Writer, [travel-sample], the view is as + have allocated Data Reader and Data Writer on + travel-sample, the view is as follows:

      - +

      @@ -173,7 +177,7 @@

      - +

      @@ -192,10 +196,7 @@

      For a complete account of the roles you can allocate and their significance, see - RBAC for - Administrators and - RBAC for - Applications. + Roles.

      @@ -213,38 +214,7 @@ user.

      -

      - This can be demonstrated with reference to the role - Cluster Admin, which is assigned by means of the Add New User - dialog. Note once again that when the Cluster Admin checkbox is manually checked, the checkboxes - for all other roles that are each considered a subset of Cluster Admin are themselves automatically - checked. -

      - -

      - -

      - -

      - When a user defined in this way logs into Couchbase Web Console, the console's appearance - is as follows: -

      - -

      - -

      - -

      - Note that the Security option has been removed from the vertical navigation-bar, at the left; since - the Cluster Admin role is not privileged to read or write security-related data. -

      - -

      - Other roles and role-combinations restrict feature-access in similar ways, and in consequence, produce different - console-appearances - for differently defined users. -

      - + diff --git a/content/security/security-rbac-upgrade.dita b/content/security/security-rbac-upgrade.dita index 7370e9e0ac..265c25d907 100644 --- a/content/security/security-rbac-upgrade.dita +++ b/content/security/security-rbac-upgrade.dita @@ -9,7 +9,7 @@ Couchbase provides an upgrade path, whereby users can seamlessly transition their activities from Couchbase Server 4.6 and earlier (which are all pre-RBAC releases) to - 5.0 (which is an RBAC-enabled release). + 5.0 and post-5.0 (which are RBAC-enabled releases). @@ -17,12 +17,12 @@
      - 5.0 Migration + Migration to 5.0 and post-5.0

      Releases of Couchbase Server prior to 5.0 did not feature the Role-Based - Access Control that is now provided. 5.0 administrators and developers must therefore + Access Control that is now provided. 5.0/post-5.0 administrators and developers must therefore become familiar with the new model, and modify procedures as appropriate.

      @@ -50,7 +50,7 @@

      Each of these types of bucket-definition is considered below, in relation to how the corresponding buckets are accessed - following upgrade to Couchbase Server 5.0. + following upgrade to Couchbase Server 5.0 and beyond.

      @@ -63,7 +63,8 @@

      Legacy buckets that were defined on the SASL-enabled standard port were protected by a - bucket-password. These bucket-passwords are no longer supported, in Couchbase Server 5.0. Instead, + bucket-password. These bucket-passwords are no longer supported, in Couchbase + Server 5.0 and beyond. Instead, buckets, as other resources, must be accessed through RBAC.

      @@ -85,9 +86,12 @@
    • - For each bucket that has been stripped of its bucket-password, a new user is created, whose username is + After all cluster-nodes have successfully + been upgraded, + for each bucket that has been stripped of its bucket-password, + a new user is created; whose username is identical to the name of the bucket. This user appears along with all others, in the Security screen - of the Couchbase Web Console. + of the Couchbase Web Console.

      @@ -114,8 +118,9 @@
    • Each new user that has been defined in this way — either with or without a password — is assigned - the Full Bucket Access role, by default. This - role allows the new user the same bucket-access privileges (including read-write on bucket-data) as were previously + the Application Access role, by default. This + role (which in RBAC-enabled versions of Couchbase Server prior to 5.5 was referred to as Full Bucket Access) + allows the new user the same bucket-access privileges (including read-write on bucket-data) as were previously possible by means of the bucket-password; and is visible in association with the user in the Security screen of the Couchbase Web Console.

      diff --git a/content/security/security-rbac-user-management.dita b/content/security/security-rbac-user-management.dita index 625ee91ed9..5a8f91c827 100644 --- a/content/security/security-rbac-user-management.dita +++ b/content/security/security-rbac-user-management.dita @@ -64,7 +64,7 @@

    • By means of the Couchbase CLI command - user-manage. + user-manage.

      @@ -106,7 +106,7 @@

    • By means of the Couchbase CLI command - user-manage. + user-manage.

      @@ -119,7 +119,12 @@

      Note that a default Couchbase password policy is provided: this can be modified by the Full Administrator. See the Couchbase CLI command - setting-password-policy. + setting-password-policy. +

      + +

      + For best practices for password-definition, see + Couchbase Passwords.

      diff --git a/content/security/security-report-vulnerability.dita b/content/security/security-report-vulnerability.dita index 68377bf5ee..42066c7025 100644 --- a/content/security/security-report-vulnerability.dita +++ b/content/security/security-report-vulnerability.dita @@ -20,11 +20,10 @@

      - To report an issue, please either - Submit a - request, - or open a JIRA issue. - Each procedure associates the issue with an identification number; which can + To report an issue, please either submit a request via the + Couchbase Support Portal, + or open a JIRA ticket. + Each procedure associates the reported issue with an identification number; which can be used for tracking purposes.

      diff --git a/content/security/security-resources-under-access-control.dita b/content/security/security-resources-under-access-control.dita index 7b14fa4b88..c6cbdbb4c5 100644 --- a/content/security/security-resources-under-access-control.dita +++ b/content/security/security-resources-under-access-control.dita @@ -28,7 +28,7 @@
      • - Clusters + Clusters.

        @@ -38,7 +38,7 @@

      • - XDCR Cluster References + XDCR Cluster References.

        @@ -48,7 +48,17 @@

      • - FTS Index Aliases + Query Service. + +

        + + +

        + +
        • + +
      • + Analytics Shadow Data Sets.

        @@ -139,7 +149,7 @@

        • - Settings: Includes
          + Settings. Includes
          Bucket Type
          @@ -183,7 +193,7 @@
        • - Statistics: Includes ops, gets, sets, and + Statistics. Includes ops, gets, sets, and deletes per second. Also includes information on memory-usage, disk-related activity, and status on indexing, querying, and XDCR activity. @@ -196,7 +206,7 @@
        • - Data: Includes data and meta-data for all objects within + Data. Includes data and meta-data for all objects within a bucket.

          @@ -218,7 +228,7 @@

        • - XDCR Bucket Replication + XDCR Bucket Replication.

          @@ -229,8 +239,50 @@

        • - Indexes. Including Views, Secondary Global Secondary - Indexes, and FTS pindexes. + Indexes. Including Views, Primary Indexes, Global Secondary + Indexes, and Search Indexes. + +

          + + +

          + +
          • + +
        • + UI Access. Allows login to Couchbase Web Console. The features available + are role-dependent. + +

          + + +

          + +
          • + +
        • + Curl Access. Allows execution of the N1QL CURL function by externally authenticated + users. + +

          + + +

          + +
          • + +
        • + Eventing. Allows configuration and scheduling of the Eventing Service. + +

          + + +

          + +
          • + +
        • + Pools.

          diff --git a/content/security/security-roles.dita b/content/security/security-roles.dita new file mode 100644 index 0000000000..7c1c08d33c --- /dev/null +++ b/content/security/security-roles.dita @@ -0,0 +1,2467 @@ + + + + + + Roles + + + + A Couchbase role permits one or more resources to be accessed according + to defined privileges. + + + + +

          + + Roles and Privileges + + +

          + Couchbase roles each have a fixed association with a set of one or + more privileges. Each + privilege is associated with a resource. Privileges are actions such as Read, Write, + Execute, Manage, Flush, and + List; or a combination of some or all of these. +

          + +

          + Roles are of two kinds: +

          + +
            +
          • + Administration and Global: Associated with cluster-wide privileges. Some of these roles are for + administrators; who might manage cluster-configurations; or read statistics; or enforce security. Others are for + users and user-defined applications that require access to specific, cluster-wide resources. +

            + +

            +
            • + +
          • + Per Bucket: Associated with one or more buckets. These roles support the reading and writing of + bucket-settings; the management of services, indexes, and replication procedures; and access to data. +

            + +

            +
            • + +
          + +

          + When an administrator, user, or application attempts to access a resource, they must authenticate, by means + of a username and password. The roles + and privileges associated with these credentials are checked + by Couchbase Server. If the associated roles contain privileges that support the kind + of access that is being attempted, access is granted; otherwise, it is denied. +

          + +

          + The following list contains all roles supported by Couchbase Server, Enterprise Edition. Each role is + explained + by means of a description and (in most cases) a table: the table lists the privileges in + association with + resources. The header of each table states the role's name, followed by its alias name + in parentheses: alias names are used in commands and queries. + In each table-body, where a privilege is associated with a resource, + this is indicated with a check-mark. Where a privilege is not associated with a + resource (or where association would not be applicable), this is indicated with + a cross. Resources not referred to in a particular table have no privileges + associated with them in the context of role being described. +

          + +

          + Note that some roles grant access to Couchbase Web Console; while others do not. The set of features + displayed within the console varies, according to role. +

          + +

          + Note also that any authentication failure will + be logged in the log file for the resource on which access was attempted. See + Logging, for detailed information + on using log files. +

          + +
          + +
          + + Full Admin + + +

          + The Full Admin role (an Administration and Global role) supports full access to all Couchbase-Server + features and resources, including those of security. The role allows access + to the Couchbase Web Console, and allows the reading and writing of bucket-data. +

          + +

          + This role is also available in Couchbase Server Community Edition. +

          + +
          + +
          + + Cluster Admin + + +

          + The Cluster Admin role (an Administration and Global role) allows the management of all cluster features + except security. + The role allows access to Couchbase Web Console, but does not permit the + writing of data. +

          + +
    + + + + + + + + + + + + + + Role: Cluster Admin (cluster_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Cluster (except Passwords) + + + + + + + + UI (except Passwords) + + + + + + + + Security (except Passwords) + + + + + + + + Bucket Data + + + + + + + + +
    + +
    + +
    + + Security Admin + + +

    + The Security Admin role (an Administration and Global role) allows the management + of user roles and the reading of all cluster statistics. The role does not permit the granting + of Full Admin or Security Admin roles, and does not permit the administrator to + change their own role (which therefore remains Security Admin). The role supports access + to Couchbase Web Console, but does not support the reading of data. +

    + + + + + + + + + + + + + + + + + Role: Security Admin (security_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Cluster + + + + + + + + UI (except Security) + + + + + + + + Security (including UI) + + + + + + + + Bucket Data + + + + + + + + +
    + +
    + + + +
    + + Read-Only Admin + + +

    + The Read-Only Admin role (an Administration and Global role) supports the reading of Couchbase clServer-statistics: + this includes registered usernames with roles and authentication domains, but excludes + passwords. + The role allows access to Couchbase Web Console. +

    + +

    + This role is also available in Couchbase Server Community Edition. +

    + + + + + + + + + + + + + + + + + Role: Read-Only Admin (ro_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Cluster + + + + + + + + UI (except Passwords) + + + + + + + + Security (except Passwords) + + + + + + + + Bucket Data + + + + + + + + +
    + +
    + + + +
    + + XDCR Admin + + +

    + The XDCR Admin role (an Administration and Global role) allows use of XDCR features, to create + cluster references and + replication streams. + The role allows access to Couchbase Web Console. +

    + + + + + + + + + + + + + + + + Role: XDCR Admin (replication_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + XDCR for Cluster and Bucket + + + + + + + + Bucket Data + + + + + + + + Bucket Settings + + + + + + + + UI (XDCR) + + + + + + + + UI (Other) + + + + + + + + +
    + +
    + +
    + + Query Curl Access + + +

    + The Query Curl Access role (an Administration and Global role) allows the N1QL CURL function + to be executed by an externally authenticated user. The user + can access Couchbase Web Console, but cannot read data, other + than that returned by the N1QL CURL function. +

    + +

    + Note that the Query Curl Access role should be assigned with caution, since + it entails risk: CURL runs within the local Couchbase Server + network; therefore, the assignee of the Query Curl Access + role is permitted + to run GET and POST requests on the internal network, while being themselves + externally located. +

    + +

    + For an account of limitations on CURL, see + CURL Function. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred to as + Query External Access. +

    + + + + + + + + + + + + + + + + + Role: Query Curl Access (query_external_access) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : N1QL, curl + + + + + + + + UI + + + + + + + + Pools + + + + + + + + +
    + +
    + +
    + + Query System Catalog + + +

    + The Query System Catalog role (an Administration and Global role) allows information to be looked up + by means of N1QL + in the system catalog: this includes system:indexes, + system:prepareds, and + tables listing current and past queries. This role is designed for troubleshooters, + who need to debug queries. The role allows access to Couchbase Web Console, but + does not permit the reading of bucket-items. +

    + + + + + + + + + + + + + + + + + + Role: Query System Catalog (query_system_catalog) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + List + + + + + + + + Bucket : N1QL, INDEX + + + + + + + + + Bucket : N1QL, Meta + + + + + + + + + UI + + + + + + + + + Pools + + + + + + + + + +
    + +

    + +

    + +
    + + +
    + + Analytics Reader + + +

    + The Analytics Reader role (an Administration and Global role) allows + querying of shadow data-sets. This + is defined as a global role because as multiple buckets may be combined into a single + shadow dataset. + The role allows access to Couchbase Web Console, and permits the reading + of data. +

    + + + + + + + + + + + + + + + + + Role: Analytics Reader (analytics_reader) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Analytics + + + + + + + + Bucket : UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + + +
    + + Bucket Admin + + +

    + The Bucket Admin role allows the management of all per bucket features + (including starting and stopping XDCR). + The role allows access to Couchbase Web Console, but does not permit the + reading or writing of data. +

    + + + + + + + + + + + + + + + + Role: Bucket Admin (bucket_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Cluster + + + + + + + + Bucket (including XDCR) + + + + + + + + Bucket Data + + + + + + + + Bucket UI + + + + + + + + Other UI + + + + + + + + +
    + +
    + + +
    + + Views Admin + + +

    + The Views Admin role allows the reading of views, per bucket. + The role allows access to Couchbase Web Console. +

    + + + + + + + + + + + + + + + + Role: Views Admin (views_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + + Bucket Data (Views) + + + + + + + + Bucket Data (Other) + + + + + + + + Bucket Settings + + + + + + + + UI (Views) + + + + + + + + UI (Other) + + + + + + + + +
    + +
    + + + + + + +
    + + Search Admin + + +

    + The Search Admin role allows management of all features of + the Search Service, per bucket. + The role allows access to Couchbase Web Console. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred to as + FTS Admin. +

    + + + + + + + + + + + + + + + + Role: Search Admin (fts_admin) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + + Bucket Data (Search) + + + + + + + + Bucket Data (Other) + + + + + + + + Bucket Settings + + + + + + + + UI (Search) + + + + + + + + UI (Other) + + + + + + + + Services and Curl + + + + + + + + Pools + + + + + + + + +
    + +
    + + + + + + + +
    + + Application Access + + +

    + The Application Access role provides read and write access to data, per bucket. + The role does not allow access to Couchbase Web Console: it is intended for + applications, rather than users. + Note that this role is also available in the Community Edition of + Couchbase Server. +

    + +

    + The role is provided in support of + buckets that were created on versions of Couchbase Server prior to 5.0. Such buckets were + accessed by specifying bucket-name and bucket-password: however, + bucket-passwords are not recognized by Couchbase Server 5.0 and after. Therefore, + for each pre-existing bucket, + the upgrade-process for 5.0 and after creates a new user, + whose username is identical to the bucket-name; and whose password + is identical to the former bucket-password, if one existed. If no bucket-password + existed, the user is created with no password. This migration-process + allows the same name-combination + as before to be used in authentication. To ensure backwards compatibility, each system-created user is + assigned the Application Access role, which authorizes the same read-write + access to bucket-data as was granted before 5.0. +

    + +

    + Use of the Application Access role is deprecated for buckets + created on Couchbase Server 5.0 and after: use the other bucket-access roles provided. + Note that in versions of Couchbase Server prior to 5.5, this role was referred to as + Bucket Full Access. +

    + + + + + + + + + + + + + + + + + + + + + Role: Application Access (bucket_full_access) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + Flush + + + + + + + + Bucket Data + + + + + + + + + + Bucket Views + + + + + + + + + N1QL: Index + + + + + + + + + N1QL: Other + + + + + + + + + Bucket + + + + + + + + + Pools + + + + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Data Reader + + +

    + The Data Reader role allows data to be read, per bucket. Note + that the role does not permit the running of N1QL + queries (such as SELECT) + against data. The role does not allow access to Couchbase Web Console: it is + intended to support applications, rather than users. +

    + + + + + + + + + + + + + + + + + Role: Data Reader (data_reader) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket Docs + + + + + + + + Bucket : Meta + + + + + + + + Bucket : Xattr + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + + +
    + + Data Writer + + +

    + The Data Writer role allows data to be written, per bucket. The role does not allow access to + Couchbase Web Console: it is + intended to support applications, rather than users. +

    + + + + + + + + + + + + + + + + + Role: Data Writer (data_writer) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Docs + + + + + + + + Bucket : Xattr + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + + +
    + + Data DCP Reader + + +

    + The Data DCP Reader role allows DCP streams to be initiated, per bucket. The + role does not allow access to Couchbase Web Console: it + is intended to support applications, rather than users. The role does + allow the reading of data. +

    + + + + + + + + + + + + + + + + + Role: Data DCP Reader (data_dcp_reader) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket: : Docs + + + + + + + + Bucket: : Meta + + + + + + + + Bucket: : DCP + + + + + + + + Bucket: : Sxattr + + + + + + + + Bucket: : Xattr + + + + + + + + Admin: Memcached: Idle + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Data Backup + + +

    + The Data Backup role allows data to be backed up and restored, per bucket. The + role supports the reading of data. The role does not allow + access to Couchbase Web Console: it is intended to support applications, + rather than users. +

    + + + + + + + + + + + + + + + Role: Data Backup (data_backup) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket: : Data + + + + + + + + Bucket: : Views + + + + + + + + Bucket: : FTS + + + + + + + + Bucket: : Stats + + + + + + + + Bucket: : Settings + + + + + + + + Bucket: : Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Data Monitor + + +

    + The Data Monitor role allows statistics to be read, per bucket. It does + not allow access to Couchbase Web Console, and does + not permit the reading of data. This role is intended to support + application-access, rather than user-access. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred + to as Data Monitoring. +

    + + + + + + + + + + + + + + + + + Role: Data Monitor (data_monitoring) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Stats + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + XDCR Inbound + + +

    + The XDCR Inbound role allows the creation of inbound XDCR streams, per bucket. It + does not allow access to Couchbase Web Console, and does not + permit the reading of data. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred to as + Replication Target. +

    + + + + + + + + + + + + + + + + + Role: XDCR Inbound (replication_target) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Settings + + + + + + + + Bucket : Meta + + + + + + + + Bucket : Stats + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Analytics Manager + + +

    + The Analytics Manager role allows management of Analytics, per bucket. + It also allows management of shadow datasets, provided that Data Read + permission has been granted on the corresponding buckets. This role allows access to Couchbase + Web Console. +

    + + + + + + + + + + + + + + + + + Role: Analytics Manager (analytics_manager) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Analytics + + + + + + + + Bucket : UI + + + + + + + + Bucket : Stats + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + + + +
    + + Views Reader + + +

    + The Views Reader role allows data to be read from the views, + per bucket. This role does not allow access to Couchbase Web + Console, and is intended to support applications, rather than + users. +

    + + + + + + + + + + + + + + + + + Role: Views Reader (views_reader) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : Docs + + + + + + + + Bucket : Views + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + + + +
    + + Search Reader + + +

    + The role Search Reader allows Full Text Search indexes to be + searched, per bucket. The role allows + access to Couchbase Web Console, and supports the reading of data. +

    + +

    + In versions of Couchbase Server prior to 5.5, this role was referred to as + FTS Searcher. +

    + + + + + + + + + + + + + + + + + Role: Search Reader (fts_searcher) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : FTS + + + + + + + + Settings: FTS + + + + + + + + UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Query Select + + +

    + The Query Select role allows the SELECT statement to be executed, + perbucket. This role allows access to Couchbase Web Console, + but does not support the reading of data. +

    + + + + + + + + + + + + + + + + + Role: Query Select (query_select) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : N1QL, SELECT + + + + + + + + UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Query Update + + +

    + The Query Update role allows the UPDATE statement to be executed, + per bucket. The role supports access to Couchbase Web Console, + but does not allow the reading of data. +

    + + + + + + + + + + + + + + + + + Role: Query Update (query_update) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : N1QL, UPDATE + + + + + + + + UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Query Insert + + +

    + The Query Insert role allows the INSERT statement to be executed, + per bucket. The role supports access to Couchbase Web Console, + but does not allow the reading of data. +

    + + + + + + + + + + + + + + + + + Role: Query Insert (query_insert) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : N1QL, INSERT + + + + + + + + UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + +
    + + Query Delete + + +

    + The Query Delete role allows the DELETE statement to be executed, + per bucket. The role supports access to Couchbase Web + Console, but does not allow the reading of data. +

    + + + + + + + + + + + + + + + + + Role: Query Delete (query_delete) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : N1QL, DELETE + + + + + + + + UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + + +
    + + Query Manage Index + + +

    + The Query Manage Index role allows indexes to be managed, + per bucket. The role allows access to Couchbase Web Console, but does + not permit the reading of data. +

    + + + + + + + + + + + + + + + + + Role: Query Manage Index (query_manage_index) + + + + Resources + Privileges + + + + Read + Write + Execute + Manage + + + + + + + + Bucket : N1QL, INDEX + + + + + + + + UI + + + + + + + + Pools + + + + + + + + +
    + +

    + +

    + +
    + + + + +
    + + System Keyspaces (Tables) + + +

    + The following system keyspaces are provided: +

    + +

    + + + + + + + System Catalogs + +
      +
    • system:datastores
      • +
    • system:namespaces
      • +
    • system:keyspaces
      • +
    • system:dual
      • +
    +     +     + + Monitoring Catalogs + +
      +
    • system:indexes
      • +
    • system:prepareds
      • +
    • system:completed_requests
      • +
    • system:active_requests
      • +
    • system:my_user_info
      • +
    +     +     + + Security Catalogs + +
      +
    • system:user_info
      • +
    • system:nodes
      • +
    • system:applicable_roles
      • +
    +     +     + + +
    +

    + SELECT Operations on System Keyspaces

    All of the system keyspaces support SELECT + operations and are divided into the below security levels:

    + + + + + diff --git a/content/security/security-saslauthd-new.dita b/content/security/security-saslauthd-new.dita index 338bc5647d..36e715db01 100644 --- a/content/security/security-saslauthd-new.dita +++ b/content/security/security-saslauthd-new.dita @@ -1,25 +1,39 @@ - Setting up <codeph>saslauthd</codeph>saslauthd is a daemon process that handles plaintext authentication requests on behalf of - the SASL library. + + + Setting up <codeph>saslauthd</codeph>saslauthd is a daemon + process that handles plaintext authentication requests on behalf of + the SASL library. + + -

    In LDAP authentication, the saslauthd process handles authentication + +

    + In LDAP authentication, the saslauthd process handles authentication requests on behalf of Couchbase Server while the LDAP protocol is used to connect to the LDAP - server.

    + server. +

    + Remote authentication with the LDAP server requires proper configuration of the saslauthd agent, which must be installed and configured on each - Couchbase Server node. - Make sure that you have the prerequisites for the LDAP software you are installing, - such as OpenLDAP libraries. - Installation packages are a part of cyrus-sasl rpm, so make - sure that it is installed. - + Couchbase Server node. + -
    Supported <codeph>saslauthd</codeph> packages for LDAP - integration +
    + + Supported <codeph>saslauthd</codeph> packages for LDAP + integration + -

    Install your Unix operating system with the package that is supported for LDAP integration.

    +

    + Install your Unix operating system with the package that is supported for LDAP integration. +

    + +

    Make sure that you have the prerequisites for the LDAP software you are installing, + such as OpenLDAP libraries. On RPM-based distros, installation packages are a part of cyrus-sasl rpm, + so make sure that it is installed.

    CentOS 6
    @@ -47,7 +61,6 @@
    saslauthd 2.1.23 or higher
    -
    @@ -55,7 +68,7 @@

    Make sure your LDAP setup is working by running a test ldapsearch as follows:

    - ldapsearch -LLL -H ldap://ldapserver:389 -D cn=someuser,ou=users,dc=mydomain,dc=com -w Passw0rd -x -bou=users,dc=mydomain,dc=com cn=someuser + ldapsearch -LLL -H ldap://ldapserver:389 -D cn=someuser,ou=users,dc=mydomain,dc=com -w Passw0rd -x -bou=users,dc=mydomain,dc=com cn=someuser

    @@ -70,107 +83,185 @@
    +
    + + + Determine mux File Location + + +

    + To communicate with the LDAP server, Couchbase Server makes use of the mux file provided + by the saslauthd package. The location of the mux file varies, according to distribution. + Couchbase Server checks for the file at two locations, which are /var/run/sasl2/mux + and /var/run/saslauthd/mux. +

    + +
      +
    • + Debian/Ubuntu: By default, the file is located at + /var/run/sasl2/mux. +

      + +

      +
      • + +
    • + RHEL/CentOS 6: By default, the file is located at + at /var/run/saslauthd/mux. +

      + +

      +
      • + + +
    • + RHEL/CentOS 7: By default, the file is located at /run/saslauthd/mux; but + a symlink from /var/run to /run/ + allows Couchbase Server to access the file at + /var/run/saslauthd/mux. +

      + +

      +
      • + +
    + +

    + If, on your system, the location of the mux file is neither + /var/run/sasl2/mux + nor /var/run/saslauthd/mux, + set the CBAUTH_SOCKPATH environment variable to the mux file's actual location. + Couchbase Server will attempt to access the mux file there. +

    + + +
    +
    - Configuring <codeph>saslauthd</codeph> Library for LDAP -

    Depending on the system, the saslauthd file is configured as follows:

    -
    - -
    Red Hat Enterprise Linux, CentOS, and Amazon Linux AMI
    -
    If you are using a system that configures saslauthd with the file - /etc/sysconfig/saslauthd, such as Red Hat Enterprise Linux, - CentOS, and Amazon Linux AMI, set the mechanism MECH to - ldap: MECH=ldap
    -     -
    -
    - -
    Ubuntu and Debian
    -
    If you are using a system that configures saslauthd with the file - /etc/default/saslauthd, such as Ubuntu, set the - MECHANISMS option to ldap: - MECHANISMS=ldap
    -
    On Debian and Ubuntu, you should also add Couchbase to the sasl group: - sudo adduser couchbase sasl
    -     -
    - -
    - -
    Configuring the <codeph>saslauthd</codeph> Configuration File -

    The default configuration file used to obtain the LDAP configuration - parameters is located at /usr/local/etc/saslauthd.conf.

    - -
    - -
    Step 1: Set up ldap_servers
    -
    Specify URIs of the LDAP servers used for authentication, such as - ldap:///10.1.1.11/, ldap://10.1.1.12/. Multiple LDAP servers - can be specified in the list, which is then tested to find out whether one of the - servers is offline. If you install OpenLDAP on the local host machine, you can specify - the value ldap://localhost:389. If using LDAP over SSL, you can specify - the value ldaps://localhost:636.
    -
    - ldap_servers: ldaps://10.1.1.25 ldaps://10.1.1.15 -
    -     -
    - -
    Step 2: Set up ldap_search_base
    -
    Specify the distinguished name to which the search is relative. The search includes - the base or objects below.
    -
    It also includes Domain Components (dc) such as in - dc=company and dc=com.
    -
    The administrative users created in LDAP with the attribute uid are + Getting Started with saslauthd and LDAP +
      +
    1. + Ensure that the Couchbase Cluster is running. Then, enable external authentication on the cluster, using the Couchbase CLI + setting-ldap command: specifying server IP-address and port number, username + and password: + + $ couchbase-cli setting-ldap -c 10.142.170.101:8091 -u Administrator -p passw0rd --ldap-enabled 1 + +

      + Note that --ldap-enabled 1 enables external authentication, and --ldap-enabled 0 disables. See + setting-ldap for further information. When + successfully executed, the command provides the following notification: SUCCESS: LDAP settings modified. +

      +
      2. + +

    2. Configure the MECHANISMS option for ldap.

      +

      Red Hat Enterprise Linux, CentOS, and Amazon Linux AMI edit + /etc/sysconfig/saslauthd (/etc/default/saslauthd on Debian/Ubuntu) + to set the mechanism MECH to + ldap:

      MECH=ldap +

      Ubuntu and Debian edit /etc/default/saslauthd, setting + MECHANISMS option to ldap:

      + MECHANISMS=ldap +

      On Debian and Ubuntu, you should also add Couchbase to the sasl group:

      + sudo adduser couchbase sasl
      3. +

    3. The default configuration file used to obtain the LDAP configuration + parameters is located at /usr/local/etc/saslauthd.conf. + Open this in your editor of choice.

      4. +

    4. Set up ldap_servers

      +

      Specify URIs of the LDAP servers used for authentication, such as + ldap:///10.1.1.11/, ldap://10.1.1.12/. Multiple LDAP servers + can be specified in the list, which is then tested to find out whether one of the + servers is offline. If you install OpenLDAP on the local host machine, you can specify + the value ldap://localhost:389.

      +

      If using LDAP over SSL, you can specify + the value ldaps://localhost:636.

      + ldap_servers: ldaps://10.1.1.25 ldaps://10.1.1.15 +
      5. +

    5. Set up ldap_search_base

      +

      Specify the distinguished name to which the search is relative. The search includes + the base or objects below.

      +

      It also includes Domain Components (dc) such as in + dc=company and dc=com.

      +

      The administrative users created in LDAP with the attribute uid are placed under the user's organizational unit ou under the two domain - components (example and com).

    -
    + components (example and com).

    +

    ldap_search_base: ou=Users,dc=company,dc=com -

    -     -
    - -
    Step 3: Set up ldap_filter
    -
    Specify the search filter. The values for these configuration options correspond to +

    + +
  • +

    Set up ldap_filter

    +

    Specify the search filter. The values for these configuration options correspond to the values specific to the test. For example, to filter on email specify - ldap_filter: (mail=%n).

    • -
    + ldap_filter: (mail=%n).

    +

    ldap_filter: (uid=%u) -

    -
    Configure LDAP options /etc/saslauthd.conf: +

    +

    Configure LDAP options /etc/saslauthd.conf: ldap_servers: ldaps://ad.example.net - ldap_search_base: ou=Users,dc=example,dc=com - ldap_filter: (uid=%u)

    -     -
    +ldap_search_base: ou=Users,dc=example,dc=com +ldap_filter: (uid=%u)

    + +

  • Running automatically

    +

    For sasld to run automatically on start up, you'll need to change the + START value to YES.

    + START = yes
    • + +

  • Test your saslauthd set-up.

    +

    If the connection is properly working, the user couchbase must have access to + /var/run/saslauthd/mux (or the appropriate alternate directory for SUSE), + in order to communicate to saslauthd.

    +
      +
    1. Start the saslauthd service (or set it to start automatically with + chkconfig).service saslauthd restart +Stopping saslauthd: [ OK ] +Starting saslauthd: [ OK ] + +chkconfig saslauthd on +chkconfig --list saslauthd +saslauthd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
      2. +
    2. Test saslauthd by using the testsaslauth script to + test LDAP authentication: + sudo -u couchbase /usr/sbin/testsaslauthd -u <username> \ +-p mypassword -f /var/run/saslauthd/mux +0: OK "Success." +
      3. +
    +
    • + +
  • + Restart the Couchbase service, to allow authentication through the changed configuration. + + $ sudo service couchbase-server restart +
    • + +
    Example - - $ cat /etc/saslauthd.conf - # ldap_servers: ldap:<URI>:<PORT> or ldaps:<URI>:<PORT> for TLS protected connection - ldap_servers: ldap://my.company.com:389 - # Specifies the distinguished name to which the search is relative. - # The search includes the base # or objects below. It also includes Domain Components (dc) - # such as in dc=company and dc=com. - # The administrative users created in LDAP with the attribute uid are placed under the user's - # organizational unit ou under the two domain components (example and com). - OU=InteractiveUsers,DC=my,DC=company,DC=com - # Specifies the search filter. The values for these configuration options correspond to the - # values specific to the test - ldap_filter: (samAccountName=%u) - # Optional: specify a user to perform ldap queries - ldap_bind_dn: CN=user_ldap,OU=Users,DC=my,DC=company,DC=com - # Optional: specify ldap user’s password - ldap_password: -sEcReTp#AssWoRd! +

    Putting the above steps into typical configuration files:

    + cat /etc/saslauthd.conf +# ldap_servers: ldap:<URI>:<PORT> or ldaps:<URI>:<PORT> for TLS protected connection +ldap_servers: ldap://my.company.com:389 +# The administrative users created in LDAP with the attribute uid are placed under the user's +# organizational unit ou under the two domain components (example and com). +OU=InteractiveUsers,DC=my,DC=company,DC=com +# Specifies the search filter. The values for these configuration options correspond to the +# values specific to the test +ldap_filter: uid=%u +# Optional: specify a user to perform ldap queries +ldap_bind_dn: CN=user_ldap,OU=Users,DC=my,DC=company,DC=com +# Optional: specify ldap user’s password +ldap_password: -sEcReTp#AssWoRd! - $ cat /etc/sysconfig/saslauthd - # Just keep the default - SOCKETDIR=/var/run/saslauthd - # Make sure MECH is set to ldap (pam is default) - MECH=ldap - # Include the config file described above - FLAGS="-O /etc/saslauthd.conf" + cat /etc/sysconfig/saslauthd +# Just keep the default +SOCKETDIR=/var/run/saslauthd +# Make sure MECH is set to ldap (pam is default) +MECH=ldap +# Include the config file described above +FLAGS="-O /etc/saslauthd.conf"
    @@ -182,41 +273,27 @@ (AD):

    ldap_servers: ldap://dc1.example.com:<port> - ldap_search_base: cn=Users,DC=ad,DC=example,DC=com - ldap_filter: sAMAccountName=%u - ldap_bind_dn: cn=saslauthd,cn=Users,DC=ad,DC=example,DC=com - ldap_password: secret +ldap_search_base: cn=Users,DC=ad,DC=example,DC=com +ldap_filter: sAMAccountName=%u +ldap_bind_dn: cn=saslauthd,cn=Users,DC=ad,DC=example,DC=com +ldap_password: secret
    - -
    Test <codeph>saslauthd</codeph> - -

    If the connection is properly working, the user couchbase must have access to - /var/run/saslauthd/mux (or the appropriate another folder for SUSE) - in order to communicate to saslauthd.

    - -
      -
    1. Start the saslauthd service (or set it to start automatically with - chkconfig).[root@localhost ~]# service saslauthd restart - Stopping saslauthd: [ OK ] - Starting saslauthd: [ OK ] - - [root@localhost ~]# chkconfig saslauthd on - [root@localhost ~]# chkconfig --list saslauthd - saslauthd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
      2. -
    2. Test saslauthd by using the testsaslauth script to - test LDAP authentication: - [root@localhost ~]# sudo -u couchbase /usr/sbin/testsaslauthd -u <username> \ --p mypassword -f /var/run/saslauthd/mux -0: OK "Success." -
      3. - -
    - - -
    - +
    + Troubleshooting LDAP Settings +

    After you set up the LDAP server, saslauthd, and LDAP administrators, likely + causes of problems include:

    +
      +
    • Firewall ports are not open for LDAP.
      • +
    • The Proxy did not start or has started with an inappropriate protocol or hostname.
      • +
    • The configuration of saslauthd is incorrect (look at /etc/sysconfig/saslauthd or + /etc/saslauthd.conf)
      • +
    • The LDAP filters are not correct.
      • +
    • You can also encounter error messages from the system. These errors belong either to issues + caused by saslauthd or the LDAP server.
      • +
    +
    diff --git a/content/security/security-session-timeouts.dita b/content/security/security-session-timeouts.dita new file mode 100644 index 0000000000..b235cc1e5e --- /dev/null +++ b/content/security/security-session-timeouts.dita @@ -0,0 +1,51 @@ + + + + + + + Session Timeouts + + + + Inactive browser-sessions should be closed. + + + +
    + + Specifying Session Timeouts + + +

    + To ensure security, inactive browser-sessions should be closed. The maximum inactivity-period is + configurable, by means of Couchbase Web Console. +

    + +

    + Access the Security panel, by means of the Security tab, at the left-hand side of + the console. Then, left-click on the Session tab, at the upper-right: +

    + +

    + +

    + +

    + The Session panel is displayed, as follows: +

    + +

    + +

    + +

    + Enter an appropriate number of minutes into the interactive field. All inactive browser-sessions + will be closed after this period. +

    + + + +
    + +     diff --git a/content/security/security-x509.dita b/content/security/security-x509.dita index 1d2a8bbd24..f17eacd75e 100644 --- a/content/security/security-x509.dita +++ b/content/security/security-x509.dita @@ -12,7 +12,7 @@

    With CA-based certificates, you also get simplified certificate management and certificate rotation without client downtime. Full administrators in Couchbase can manage certificates using the Couchbase CLI tools (as described in ) or REST SPI (as described in ) or REST SPI (as described in ).

    Your prior knowledge of TLS/SSL, PKI certificates including X.509 certificates, and Certificate Authorities (CAs) is assumed for this overview. diff --git a/content/security/security-x509certsintro.dita b/content/security/security-x509certsintro.dita index 6655af6bb4..fb280ca692 100644 --- a/content/security/security-x509certsintro.dita +++ b/content/security/security-x509certsintro.dita @@ -25,7 +25,7 @@

    Full administrators can manage certificates, using the Couchbase CLI - tools (as described in ) and REST + tools (as described in ) and REST API (as described in ).

    @@ -372,8 +372,8 @@

    - The following steps show how to configure X.509 certificates on Ubuntu 16: a root certificate is - created with a single intermediate certificate and a single node certificate; and a chain-certificate + The following steps configure X.509 certificates on Ubuntu 16: a root certificate is + created with a single intermediate certificate and a single node certificate; and a chain certificate is created from the intermediate and node certificates. The chain certificate and node private key are then made active for the current Couchbase Server-node.

    @@ -417,7 +417,7 @@ export CHAIN=chain
  • - Create environment variables for the administrator-credentials to be used for certificate-management, + Create environment variables for the administrator-credentials to be used for certificate management, the IP address at which the Couchbase Server-node is located, and the username required for role-based access to a particular resource. @@ -489,7 +489,7 @@ openssl req -new -key ${INTERMEDIATE}.key -out ${INTERMEDIATE}.csr \
  • - Create the extension file v3_ca.ext; in order to add extensions to the + Create the extension-file v3_ca.ext; in order to add extensions to the certificate, and to generate the certificate signing-request: cat <<EOF>> ./v3_ca.ext @@ -595,17 +595,11 @@ couchbase-cli ssl-manage -c ${ip}:8091 -u Administrator -p password \
  • For the current Couchbase Server-node, enable the client certificate: - curl -X POST --data-binary "state=enable" \ -http://${ADMINCRED}@${ip}:8091/settings/clientCertAuth -curl -X POST --data-binary "delimiter=" \ -http://${ADMINCRED}@${ip}:8091/settings/clientCertAuth -curl -X POST --data-binary "path=subject.cn" \ -http://${ADMINCRED}@${ip}:8091/settings/clientCertAuth -curl -X POST --data-binary "prefix=" \ -http://${ADMINCRED}@${ip}:8091/settings/clientCertAuth + curl -u ${ADMINCRED} -v -X POST http://${ip}:8091/settings/clientCertAuth \ +-d '{"state": "enable","prefixes": [{"path": "subject.cn","prefix": "","delimiter": ""}]}'

    For further information on certificate-deployment, see - ssl-manage and + ssl-manage and Encryption On-the-Wire API.

    • @@ -621,7 +615,8 @@ http://${ADMINCRED}@${ip}:8091/settings/clientCertAuth

    Once the root certificate for a Couchbase Server-node has been deployed, a Java client can - authenticate by means of an appropriately prepared keystore. + authenticate by means of an appropriately prepared keystore, and so gain access to the + Data, Query, and Search Services.

    @@ -727,6 +722,8 @@ export STOREPASS=storepass This concludes preparation of the Java client's keystore. Copy the file (in this case, my.keystore) to a location on a local filesystem from which the Java client can access it.

    + + diff --git a/content/security/security.ditamap b/content/security/security.ditamap index 5d74a9a667..a65fd0df15 100644 --- a/content/security/security.ditamap +++ b/content/security/security.ditamap @@ -8,8 +8,6 @@ - - @@ -18,7 +16,6 @@ - @@ -28,12 +25,13 @@ - - - + + + + - + @@ -69,8 +67,8 @@ - + diff --git a/content/server-architecture.ditamap b/content/server-architecture.ditamap index 0cda2166cf..de82a32fbd 100644 --- a/content/server-architecture.ditamap +++ b/content/server-architecture.ditamap @@ -12,6 +12,8 @@ + + diff --git a/content/settings/backfill.dita b/content/settings/backfill.dita new file mode 100644 index 0000000000..c5573dab47 --- /dev/null +++ b/content/settings/backfill.dita @@ -0,0 +1,93 @@ + + + + Backfill Support for N1QL + Configure the temporary working space for the N1QL engine and its embedded GSI client. + +

    (Introduced in Couchbase Server 5.5)

    +
    + Introduction +

    When a query has an extremely large corresponding index scan, the indexer + gsiClient buffers the results into the /tmp + directory. Since this method has caused some high I/O and works differently on + Windows, Couchbase Server 5.5 now lets the user configure the backfill directory via + the UI, which will then pass the settings to the server.

    +
    +
    + Prerequisites +

    The user must have Cluster Admin privileges in order to be able to specify and + configure the location for temporarily storing any transient-data during query + processing per query-engine.

    +
    +
    Configuration

    In the Query Workbench > + Settings page, click Advanced Query Settings to see:

    In the Query Temp Disk Path textbox, enter the directory you want to use + for your backfill file.

    In the Quota textbox, enter the number of megabytes for + the maximum size of your backfill file.

    Your system administrator can read and + modify the two REST backfill parameters: + + + + + + + + Setting + Description + Values + Remarks + + + + + query_tmpspace_dir + The temp directory in which the indexer can create + the scan_backfill file. + Default value is /tmp + Only absolute paths are allowed. + + + query_tmpspace_limit + The maximum limit in MB for the temporary file size. + 0 will disable + backfill.

    -1 means the + size is unlimited.

     + The max size is limited only by your available disk + space. +     + + +

    +
    + Examples +

    +

      +
    • Ex 1: Curl request to get the + current temp file size.
      • +
    • Ex 2: Curl request to change the + temp file directory to tmp2
      • +
    • Ex 3: Curl request to change the + temp file size to 2048 MB.
      • +
    +

    +

    Example 1: Curl request to get the current "Advanced Query + Settings".curl --GET -u Administrator:password http://localhost:8091/settings/querySettings Results:{"queryTmpSpaceDir":"/opt/couchbase/tmp","queryTmpSpaceSize":5120,"queryCurlWhitelist":{"all_access":false,"allowed_urls":["http://localhost:8091/settings/querySettings",""],"disallowed_urls":[""]}}Example 2: Curl request to change the temp file directory to + tmp2 (after making a tmp2 + directory).curl -X POST -u Administrator:password -d 'queryTmpSpaceDir=/opt/couchbase/tmp2' http://localhost:8091/settings/querySettingsResults:{"queryTmpSpaceDir":"/opt/couchbase/tmp2","queryTmpSpaceSize":5120,"queryCurlWhitelist":{"all_access":false,"allowed_urls":["http://localhost:8091/settings/querySettings",""],"disallowed_urls":[""]}}Example 3: Curl request to change the temp file size to 2048 + MB.curl -X POST -u Administrator:password -d 'queryTmpSpaceDir=/opt/couchbase/tmp' -d 'queryTmpSpaceSize=2048' http://localhost:8091/settings/querySettingsResults:{"queryTmpSpaceDir":"/opt/couchbase/tmp","queryTmpSpaceSize":2048,"queryCurlWhitelist":{"all_access":false,"allowed_urls":["http://localhost:8091/settings/querySettings",""],"disallowed_urls":[""]}}At + the same time, the logs will contain messages for every change coming from both + query and GSI, so after executing Example 3's CURL + statement:_time=2017-12-06T14:12:39.709-08:00 _level=INFO _msg=New settings received: {"query.settings.tmp_space_dir":"/opt/couchbase/tmp/","query.settings.tmp_space_size":2048} + +_time=2017-12-06T14:12:39.712-08:00 _level=INFO _msg=Settings notifier from metakv + +2017-12-06T14:12:39.715-08:00 [Info] GSIC - Setting config map[query_tmpspace_dir:/opt/couchbase/tmp/ query_tmpspace_limit:2048] + +_time=2017-12-06T14:12:39.715-08:00 _level=INFO _msg= Indexer settings have been updated map[query_tmpspace_dir:/opt/couchbase/tmp/ query_tmpspace_limit:2048]

    +
    + +     diff --git a/content/settings/change-failover-settings.dita b/content/settings/change-failover-settings.dita index 5c59df79f5..54a8b11fdc 100644 --- a/content/settings/change-failover-settings.dita +++ b/content/settings/change-failover-settings.dita @@ -1,44 +1,127 @@ - Define Failover SettingsWhen enabled, the auto-failover feature will automatically failover a node identified as - unresponsive or unavailable. + + + Node Availability + + + + When enabled, auto-failover automatically fails over any node identified as + unresponsive or unavailable. + + -

    Full Administrators and Cluster Administrators can enable auto-failover feature.

    -

    Automatic failover is disabled by default. When you enable the auto-failover, it is enabled - for all services except for the Index Service. The timeout settings determine how long a node - is unresponsive to the rest of the cluster before the auto-failover process triggers a - failover.

    + +

    + Only Full Administrators and Cluster Administrators can enable or disable auto-failover. + Auto-failover is disabled by default. When enabled, auto-failover handles all + services except the Index Service. +

    +
    - Using the UI -

    -

    To enable the automatic failover using the Web Console:

    -
      -
    1. From the Couchbase Web Console > Settings > - select the Auto-Failover tab.
      2. -
    2. Select the Enable auto-failover check box to enable the - setting.
      3. -
    3. To set the delay in seconds before auto-failover is started, enter the number of seconds - in the Timeout box and click Save. The default - timeout is 120 seconds. In almost all cases, it is recommended to leave the timeout at the - default.
      4. -
    + + + Using the UI + + +

    + The Node Availability panel is as follows: +

    + +

    + +

    + +

    + The following three checkboxes are provided: +

    + +
      +
    • + Enable auto-failover after x seconds for up to y event: After the + timeout period set here as x seconds has elapsed, and up to the limit of actionable events set here + as y, an unresponsive or malfunctioning node + is failed over. Replica copies of data, indexes, or query engines are promoted to active on other nodes, + as appropriate. Note that this feature can only used when three or more nodes are present in the cluster. See + Using Automatic Failover + for more information. + +

      + +

      +
      • + +
    • + Enable auto-failover for sustained data disk read/write failures after z seconds: After the + timeout period set here as z seconds has elapsed, a node + is failed over if it has experienced sustained data disk read/write failures. This checkbox can only + be checked if Enable auto-failover after n seconds for up to x event has + also been checked. + +

      + +

      +
      • + +
    • + Enable auto-failover for server groups: Server-group failover is enabled. This checkbox (which can only be + checked if + Enable auto-failover after n seconds for up to nn event + has also been checked) + should only be checked + if three or more server groups have been established, and capacity is available to absorb the combined load of + all potentially + failed-over groups. For information on groups and Server Group Awareness, see + Groups. + +

      + +

      +
      • + +
    + +

    + For more information on auto-failover, see + Using Automatic Failover. +

    + +

    + The Node Availability screen contains the following, additional option, which is is available for Ephemeral Buckets: +

    + + +

    + +

    + +

    + Checking this checkbox ensures that if a node containing active Ephemeral Buckets becomes unavailable, its + replicas on other nodes are promoted to active status as appropriate, to avoid data-loss. Note, however, that this may leave the cluster + in an unbalanced state, requiring a rebalance. +

    +
    Using the CLI -

    The automatic failover settings can be changed using the setting-autofailover command in couchbase-cli.

    +

    The automatic failover settings can be changed using the + setting-autofailover command in couchbase-cli.

    Using the REST API -

    The automatic failover settings can be changed using the REST API /settings/autoFailover endpoint.

    -

    Below is an example of changing the automatic failover settings using the REST API:

    +

    The automatic failover settings can be changed using the REST API + /settings/autoFailover endpoint.

    +

    + Below is an example of changing the automatic failover settings using the REST API: +

    +
    - +     diff --git a/content/settings/cluster-settings.dita b/content/settings/cluster-settings.dita index da5f40ffe3..959bafc4d3 100644 --- a/content/settings/cluster-settings.dita +++ b/content/settings/cluster-settings.dita @@ -1,19 +1,27 @@ - Configure Cluster SettingsThe Settings menu sets the global settings for your Couchbase - Server instance. + ClusterThe Cluster panel can be used to establish settings for the + cluster. -

    Full Administrators and Cluster Administrators can configure cluster settings.

    +

    + Only Full Administrators and Cluster Administrators can configure cluster settings.

    Use the Settings > Cluster tab to configure the cluster name, memory - quotas, and index settings. You can define the settings using the Web Console, with the CLI - command, or using the REST API.

    -

    To configure cluster settings using the Web Console:

    + quotas, and index settings. You can define the settings using Couchbase Web Console, the CLI, or the REST API. +

    + +

    + +

    + +

    + To configure cluster settings using Couchbase Web Console: +

    +
    1. Select the Couchbase Web Console > Settings menu. By default, the Cluster page is displayed.
      2. -
    2. Enter the following settings and click Save:
      3. +
    3. Enter appropriate information into the fields listed below, and click Save:
    Cluster Name @@ -59,23 +67,56 @@ of all buckets combined. - GSI Service + Index Service This setting controls the buffer cache size of the index storage layer. The specified memory is pre-allocated as soon as the indexer starts up, and is shared with all indexes created on the node. The total memory usage of the indexer process will be the buffer cache plus the size of various internal data structures and queues. + - FTS Service + Search Service This setting controls the RAM quota for Full Text search. It shows the amount of the main memory allocation for Full Text service per node. + + + Analytics Service + This setting controls the RAM quota for Analytics. It shows the amount + of the main memory allocation for the Analytics service per node. + + + + Eventing Service + This setting controls the RAM quota for Events. It shows the amount + of the main memory allocation for the Events service per node. + +
    + + +
    + + + Advanced Query Settings + + +

    + Specify either Unrestricted or Restricted, to determine which URLs are permitted to be + accessed by the curl function. If Unrestricted (the default) is specified, all + URLs can be accessed. If Restricted is specified, the UI expands, to display configurable fields + into which the URLs allowed and disallowed can be entered. +

    + +
    + + +
    Advanced Index Settings for Standard and Memory-Optimized Indexes

    Define the Global Secondary Index type as Standard or Memory-Optimized option and then click Show Advanced Index Settings pull-down menu.

    @@ -115,9 +156,23 @@
    + +
    + + + Query Temp Disk Path + + +

    + The path to which temporary files are written, based on query activities. Specify a maximum size in + megabytes. +

    + +
    +

    - +     diff --git a/content/settings/configure-alerts.dita b/content/settings/configure-alerts.dita index 381fae66af..735efc3377 100644 --- a/content/settings/configure-alerts.dita +++ b/content/settings/configure-alerts.dita @@ -1,31 +1,51 @@ - Configure Email AlertsWhen alerts are configured, which raise email alerts when a significant error occurs on - the Couchbase Server cluster. + + + Email Alerts + + + + Email alerts can be dispatched automatically when a significant error occurs. + + -

    Full Administrators and Cluster Administrators can configure alerts.

    -

    The email alert system works by sending an email directly to a configured SMTP server. Each - alert email is sent to the list of configured email recipients to highlight specific issues - and problems that you should be aware of and may need to check to ensure the health of your - Couchbase cluster. Alerts are also provided as a popup within the Couchbase Web Console.

    -

    It is important to ensure that all Couchbase Server nodes have network access to the configured SMTP server. +

    + Only Full Administrators and Cluster Administrators can configure alerts.

    -

    You can define the settings using the Web Console or using the + The email-alert system works by sending an email directly to a configured SMTP server. Each + alert email is sent to the configured list of email recipients, to highlight specific issues + and problems. Alerts are provided as a popup within Couchbase Web Console. +

    + +

    + It is important to ensure that all Couchbase Server nodes have network access to the configured SMTP server. +

    + +

    + You can define the settings using Couchbase Web Console, or using the REST - API.

    -

    -

    To setup email alerts using the Web Console:

    -
      -
    1. From the Couchbase Web Console > Settings > - select the Email Alerts tab.
      2. -
    2. Select the Enable email alerts check box to enable the email alerts - configuration. These alerts are raised for errors that you select in the Available - Alerts section.
      3. -
    3. Define the following settings and click Save:
      4. -
    + API. +

    + +

    + The Email Alerts screen appears as follows: +

    + +

    + +

    + +

    + To set up email alerts using Couchbase Web Console, first, select the Enable email alerts check box, to enable the email-alerts + configuration. Then, enter appropriate data into the fields listed below. +

    +
    Email Server Settings @@ -104,88 +124,115 @@
    Available Alerts -

    You can enable individual alert messages that can be sent by selecting the appropriate - check boxes. The supported alerts are:

    +

    + You can enable individual alert messages that can be sent by selecting the appropriate + check boxes. The supported alerts are: +

    + - + + Alert Description + Code + Node was auto-failed-over The sending node has been failed over automatically. + auto_failover_node Maximum number of auto-failed-over nodes was reached The auto-failover system stops auto-failover when the maximum number of spare nodes available has been reached. + auto_failover_maximum_reached Node wasn't auto-failed-over as other nodes are down at the same time Auto-failover does not take place if there is already a node down. + auto_failover_other_nodes_down Node was not auto-failed-over as there are not enough nodes in the cluster running the same service You cannot support auto-failover with less than three nodes. + auto_failover_cluster_too_small Node was not auto-failed-over as auto-failover for one or more services running on the node is disabled Auto-failover does not take place on a node as one or more services running on the node is disabled. + auto_failover_disabled Node's IP address has changed unexpectedly The IP address of the node has changed, which may indicate a network interface, operating system, or other network or system failure. + ip Disk space used for persistent storage has reach at least 90% of capacity The disk device configured for storage of persistent data is nearing full capacity. + disk Metadata overhead is more than 50% The amount of data required to store the metadata information for your dataset is now greater than 50% of the available RAM. + overhead Bucket memory on a node is entirely used for metadata All the available RAM on a node is being used to store the metadata for the objects stored. This means that there is no memory available for caching values. With no memory left for storing metadata, further requests to store data will also - fail.

    Only applicable to buckets with Value-only ejection policy.

    + fail. + +

    + Only applicable to buckets configured for + value-only ejection. See + Ejection, + for information. +

    + + + ep_oom_errors     Writing data to disk for a specific bucket has failed The disk or device used for persisting data has failed to store persistent data for a bucket. + ep_item_commit_failed Writing event to audit log has failed The audit log event writing has failed. + audit_dropped_events Approaching full Indexer RAM warning The indexer RAM limit threshold is approaching warning. + indexer_ram_max_usage Remote mutation timestamp exceeded drift threshold The remote mutation timestamp exceeded drift threshold warning. + ep_clock_cas_drift_threshold_exceeded Communication issues among some nodes in the cluster There are some communication issues in some nodes within the cluster. + communication_issue @@ -193,6 +240,6 @@ - + diff --git a/content/settings/configure-compact-settings.dita b/content/settings/configure-compact-settings.dita index 374c137d82..8425b84301 100644 --- a/content/settings/configure-compact-settings.dita +++ b/content/settings/configure-compact-settings.dita @@ -2,7 +2,7 @@ - Configuring Auto-Compaction + Auto-Compaction @@ -18,7 +18,7 @@

    Auto-Compaction is enabled by default for all Couchbase buckets. However, settings - (with the exception of those for index fragmentation) can + can be overridden on a per bucket basis.

    @@ -27,63 +27,63 @@ To access the Auto-Compaction settings:

    -
      - -
    • - For a new bucket: Access the Buckets - screen provided by the Couchbase Web Console. Left-click on - the Add Bucket button: - -

      - -

      - -

      - When the Add Data Bucket - dialog appears, add appropriate data into the initial fields; then left-click on the - Advanced bucket settings tab: -

      - -

      - -

      - -

      +

        + +
      • + For a new bucket: Access the Buckets + screen provided by the Couchbase Web Console. Left-click on + the Add Bucket button: + +

        + +

        + +

        + When the Add Data Bucket + dialog appears, add appropriate data into the initial fields; then left-click on the + Advanced bucket settings tab: +

        + +

        + +

        + +

        The dialog now expands, and displays additional configuration-options. Access the Auto-Compaction panel, and check the Override the - default auto-compaction settings? checkbox: -

        - -

        - -

        - -

        - The dialog expands again, and displays Auto-Compaction settings available for this bucket. -

        - -
        • - -
      • - For an existing bucket: Left-click on the information-row - for the bucket, on the Buckets screen of the Couchbase Web Console. When - the Edit button appears, left-click on it: - -

        - -

        - -

        - This brings up the Edit + default auto-compaction settings? checkbox: +

        + +

        + +

        + +

        + The dialog expands again, and displays Auto-Compaction settings available for this bucket. +

        + +
        • + +
      • + For an existing bucket: Left-click on the information-row + for the bucket, on the Buckets screen of the Couchbase Web Console. When + the Edit button appears, left-click on it: + +

        + +

        + +

        + This brings up the Edit Bucket Settings dialog: left-click on the Show advanced bucket - settings tab. Access - the Auto-Compaction panel, and check the Override the + settings tab. Access + the Auto-Compaction panel, and check the Override the default auto-compaction settings? checkbox. The dialog expands, thereby showing - the available Auto-Compaction settings. -

        + the available Auto-Compaction settings. +

        • - +
      • For all buckets for which no override is specified: Left-click on the Settings tab, in the vertical navigation-bar at the left-hand side. When the Settings screen appears, left-click @@ -98,7 +98,7 @@

        • - +
      @@ -111,9 +111,9 @@

      The Auto-Compaction view of the Settings screen appears - as follows: + as follows.

      - +

      @@ -121,16 +121,14 @@

      All settings on this screen are also provided on the dialogs whereby you establish custom-settings for an individual new or - existing bucket — with the exception of the settings for index fragmentation, which can only be - established on - a cluster-wide basis. + existing bucket.

      Settings constitute conditions, which must be met for the compaction-process to be triggered. The settings are described below.

      - +
      @@ -163,7 +161,7 @@

      The View Fragmentation panel appears as follows:

      - +

      @@ -216,87 +214,14 @@
      -
      - - - Index Fragmentation - - -

      - The Index Fragmentation panel provides settings that cannot be overridden at - individual bucket-level. The panel appears as follows: -

      - -

      - -

      - -

      - This interface sets the write-strategy and trigger-point for compaction. - Note that this option - only applies when the Standard Global Secondary Index storage option is selected for - indexes. Note also that write mode and compaction strategy does not apply to memory-optimized global - secondary indexes. -

      - -

      - Select from the following options: -

      - -
        - -
      • - Append-only write mode with index fragmentation level trigger: - -

        - Turns on append only writes for index-storage, and triggers the compaction-job based - on the fragmentation-level of each index file. Check the checkbox, then specify a fragmentation-level - as a percentage, in the interactive text-field. -

        - -

        - -

        - -
        • - -
      • - Circular write mode with day + time interval trigger - -

        - Turns on writes with - circular reuse, - for index-storage, and triggers the compaction-job based on a - time-interval. To specify when compaction is permitted to run, select - appropriate days of the week, by checking the appropriate checkboxes; then, select the start-time - on each of those days; and optionally, an end-time. -

        - -

        - Optionally, check the Abort compaction if run time - exceeds the set time interval checkbox: if you do so, compaction is aborted if the specifed - end-time is exceeded. -

        - -
        • - -
      - -

      - Note that whenever you change the compaction settings for the index, the system starts the - global secondary index process on all the nodes. -

      - -
      -
      - Metadata Purge Interval + Tombstone Purge Interval

      - Sets the frequency of metadata purge interval. The default value is three days. The panel appears as follows: + Sets the frequency of the tombstone (metadata) purge interval. The default value is three days. The panel appears as follows:

      @@ -308,7 +233,10 @@ key and metadata. Tombstones are used in Couchbase Server to provide eventual consistency of data between clusters. The auto-compaction process waits for the specified number of days before permanently deleting tombstones for expired or deleted - items. + items. The default value is three days. The recommended range of values is 0.04 to + 60 (where + 0.04 equals one hour, and + 1 equals one day.

      @@ -320,7 +248,7 @@

      For more information, see - . + Storage.

      @@ -336,12 +264,12 @@

      - + - - + + diff --git a/content/settings/install-sample-buckets.dita b/content/settings/install-sample-buckets.dita index 1709604035..a3af685274 100644 --- a/content/settings/install-sample-buckets.dita +++ b/content/settings/install-sample-buckets.dita @@ -1,34 +1,61 @@ - Install Sample BucketsThe Sample Buckets tab allows you to install the sample bucket - data if it has not already been loaded in the system. + + Sample Buckets + + + + Sample Buckets contain data that is ready to be experimented with. + - Remove the default bucket and sample buckets in production. Both types of - buckets are created without a password protection and if permissions to such buckets are later - assigned to users they will become "orphaned" in case these buckets are deleted. - +

      + Only Full Administrators and Cluster Administrators can install sample buckets. +

      + +

      + The Sample Buckets screen appears as follows: +

      + +

      + +

      + +

      + To install sample buckets: +

      -

      Full Administrators and Cluster Administrators can install sample buckets.

      -

      The sample buckets are located in the Couchbase installation directory as .zip files as well. - On Linux, the default location would be /opt/couchbase/samples. The - sample bucket Zip files can be loaded into the cluster by using the cbdocloader tool. Example: - /opt/couchbase/bin/cbdocloader -n localhost:8091 -u Admin -p Pass -b mybucket -s 100 /opt/couchbase/samples/travel-sample.zip

      -

      -

      To enable the update notifications using the Web Console:

        -
      1. From the Couchbase Web Console > Settings > - select the Sample Buckets tab.
        2. -
      2. Select the sample buckets that you want to load using the check boxes and click - Load Sample Data.

        If the sample bucket data has already been - loaded, it is listed under the Installed Samples section of the page. -

        3. +
      3. + From the Couchbase Web Console > Settings > + select the Sample Buckets tab. +
        4. + +
      4. + Select the sample buckets that you want to load using the check boxes and click + Load Sample Data. +
      + +

      + If the sample bucket data has already been + loaded, it is listed under the Installed Samples section of the page. +

      + +

      + Note that as well as being accessible from the Sample Buckets screen, sample buckets are also available + as zip files, in + the Couchbase installation directory (whose default location + on Linux would be /opt/couchbase/samples. The + zip files can be loaded by means of the + cbdocloader tool. For example: +

      + + /opt/couchbase/bin/cbdocloader -n localhost:8091 -u Admin -p Pass -b mybucket -s 100 /opt/couchbase/samples/travel-sample.zip + + + - - - +       diff --git a/content/settings/pict/backfill_QueryTempDiskPath_circled.png b/content/settings/pict/backfill_QueryTempDiskPath_circled.png new file mode 100644 index 0000000000..c306590fef Binary files /dev/null and b/content/settings/pict/backfill_QueryTempDiskPath_circled.png differ diff --git a/content/settings/pict/cluster-settings.png b/content/settings/pict/cluster-settings.png new file mode 100644 index 0000000000..5fbf93e118 Binary files /dev/null and b/content/settings/pict/cluster-settings.png differ diff --git a/content/settings/query-settings.dita b/content/settings/query-settings.dita index 35e2b54bb0..192c3254c5 100644 --- a/content/settings/query-settings.dita +++ b/content/settings/query-settings.dita @@ -283,9 +283,9 @@ curl http://hostname:8093/admin/settings -d '{"controls":true include this parameter.

      If a request includes max_parallelism, it will be capped by the server max_parallelism.

      - cbq> \set -max-parallelism 3; + cbq> \set -max_parallelism 3; -curl http://hostname:8093/admin/settings -d '{"max-parallelism":0}' -u user:pword +curl http://hostname:8093/admin/settings -d '{"max_parallelism":0}' -u user:pword

      @@ -331,10 +331,9 @@ curl http://hostname:8093/admin/settings -d '{"pipeline-cap": prepared-limit int 16384 - The number of bytes the Prepared Limit sizes the - prepared statement cache.

      When this cache reaches the limit, the least recently - used prepared statements will be discarded as new prepared statements are - created.

      + [Optional] Maximum number of Prepared statements + in the cache.

      When this cache reaches the limit, the least recently used prepared + statements will be discarded as new prepared statements are created.

      curl http://hostname:8093/admin/settings -d '{"prepared-limit":65536}' -u user:pword

      diff --git a/content/settings/settings-ldap.dita b/content/settings/settings-ldap.dita index a49fb760d9..b76018a7bf 100644 --- a/content/settings/settings-ldap.dita +++ b/content/settings/settings-ldap.dita @@ -13,6 +13,6 @@ href="../security/security-ldap-new.dita#topic_bgy_3ng_tq">LDAP authentication.

      - + diff --git a/content/settings/update-notificatioin.dita b/content/settings/update-notificatioin.dita index 7bb12e8696..fd27cf0d6a 100644 --- a/content/settings/update-notificatioin.dita +++ b/content/settings/update-notificatioin.dita @@ -1,49 +1,64 @@ - Update Product NotificationsThe Software Updates page allows the client to determine whether a - newer version of Couchbase Server is available. + + + Software Updates + + + + The Software Updates page allows notifications to be provided as to whether a + newer version of Couchbase Server is currently available. + + -

      Full Administrators and Cluster Administrators can update notifications function.

      -

      During installation, you can choose to enable the Update Notification - function. After installation, you can also enable or disable the setting from the +

      + During installation, the Update Notification option can be optionally enabled. + After installation, Full Administrators and Cluster Administrators can enable or disable the setting from Couchbase Web Console > Settings > - Software Updates tab.

      -

      -

      To enable the update notifications using the Web Console:

      -
        -
      1. From the Couchbase Web Console > Settings > - select the Software Updates tab.
        2. -
      2. Select the Enable software update notificationscheck box to enable - the updates and click Save.
        3. -
      -

      If update notifications are disabled, then the Software Updates page - only notifies you of your currently installed version, and no alert is provided.

      -

      If you enable the update notifications, the Couchbase Web Console communicates with Couchbase + Software Updates tab. +

      + +

      + +

      + +

      + To enable or disable the update notifications option, check or uncheck the Enable software update notifications + checkbox. +

      + +

      + If update notifications are disabled, the Software Updates page + only notifies you of your currently installed version, and no alert is provided. +

      + +

      + If you enable the update notifications, Couchbase Web Console communicates with Couchbase Server to confirm the version number of your Couchbase Cluster. During this process, the - client also submits the following information back to Couchbase Server:

      + client also submits the following information to Couchbase Server: +

      +
      • The current version of your Couchbase Server installation.
        • -
      • Basic information about the data size and performance.
        • -
      • The configuration of your Couchbase cluster such as which features are used.
        • +
      • Basic information about the data-size and performance.
        • +
      • The configuration of your Couchbase cluster, including which features are used.

      This information is used to help Couchbase prioritize development efforts.

      -

      The update notification communication to Couchbase occurs in the browser accessing the web - console, not from the Couchbase cluster itself. No further configuration or internet access is - required in the Couchbase Server to enable this functionality. The update notification process - works anonymously, and the data cannot be tracked. Identifiable information such as bucket - names, bucket data, design document names and hostnames are not transmitted.

      -

      If an update is available, the notification will look like the following in the Couchbase - Web Console:

      -

      - If the browser or the machine you are using to connect to your Couchbase Web - Console does not have Internet access, the update notification system does not work. +

      The communication to Couchbase occurs from the browser accessing the web + console, not from the Couchbase cluster itself. No further configuration of + Couchbase Server is required to enable this functionality. The update-notification process + works anonymously: data cannot be tracked. Identifiable information (such as bucket + names, bucket data, design-document names, and hostnames) is not transmitted. +

      + +

      + Note that if the browser or the machine you are using to connect to your Couchbase Web + Console does not have internet access, the update-notification system does not work. +

      + - - - +       diff --git a/content/third-party-licenses.dita b/content/third-party-licenses.dita index dffd4da551..4aa5658ae2 100644 --- a/content/third-party-licenses.dita +++ b/content/third-party-licenses.dita @@ -4,10 +4,11 @@ Licenses for Third-Party Components

      This section contains licensing information for some third-party components that are used by - Couchbase Server. The complete list of third-party licenses for Couchbase Server is available - on the Couchbase Server Third Party Licenses page. We are thankful to all - individuals that have created these components.

      + Couchbase Server. We are thankful to all + individuals that have created these third-party components.

      +

      IMPORTANT: The complete list of licenses for Couchbase Server is available + on the Legal Agreements page.

      PCRE LICENCE

      PCRE is a library of functions to support regular expressions whose syntax and semantics diff --git a/content/tools/cbimport.dita b/content/tools/cbimport.dita index 7bc288a6b6..25b4976025 100644 --- a/content/tools/cbimport.dita +++ b/content/tools/cbimport.dita @@ -101,9 +101,9 @@ -d,--dataset <uri> The URI of the dataset to be loaded. cbimport supports - loading data from a local file or from a URL. When importing data from a local - file the path must be prefixed with file://. If a URL is used - then the file should be prefixed with either http:// or + loading data from a local file or from an internet location. When importing data from a local + file the path must be prefixed with file://. If an internet location is used + then the path should be prefixed with either http:// or https://. @@ -314,9 +314,9 @@ barry 36 -d,--dataset <uri> The URI of the dataset to be loaded. cbimport supports - loading data from a local file or from a URL. When importing data from a local - file the path must be prefixed with file://. If a URL is used - then the file should be prefixed with either http:// or + loading data from a local file or from an internet location. When importing data from a local + file the path must be prefixed with file://. If an internet location is used + then the path should be prefixed with either http:// or https://. @@ -420,8 +420,7 @@ barry 36

    • Sample

      The sample format specifies a ZIP file or folder containing multiple documents. This format is intended to load Couchbase sample data sets. Unlike the lines and list formats, the sample format may also contain index, view, and full-text - index definitions. If using this format, you should not specify the path should not include - the file:// prefix. Here's the folder structure for the sample format: + index definitions. Here's the folder structure for the sample format: + (root folder) + docs key1.json diff --git a/content/tools/cbq-shell.dita b/content/tools/cbq-shell.dita index a96deef031..49956c838b 100644 --- a/content/tools/cbq-shell.dita +++ b/content/tools/cbq-shell.dita @@ -88,7 +88,7 @@ $ ./cbq --help Use the \HELP - +

    Option @@ -111,10 +111,12 @@ $ ./cbq --help Use the \HELP couchbases:// protocol schemes. When using the couchbase:// or couchbases:// protocol schemes, you need not specify the port when connecting to the Couchbase cluster. -

    Shell command:

    The cbq shell supports both IPV4 and IPV6 addresses.

    Shell command: + \CONNECT

    Examples

    $ ./cbq -e couchbase://localhost

    - $ ./cbq --engine http://localhost:8093 + $ ./cbq --engine http://localhost:8091 $ ./cbq -e http://localhost:8091 + $ ./cbq -e http://[fd63:6f75:6368:1075:816:3c1d:789b:bc4]:8091 Connected to : http://localhost:8091/. Type Ctrl-D or \QUIT to exit. Path to history file for the shell : /Users/myuser1/.cbq_history cbq> @@ -284,7 +286,7 @@ $ ./cbq -skip-verify https://127.0.0.1:18091 - +

    Shell Command @@ -304,8 +306,9 @@ $ ./cbq -skip-verify https://127.0.0.1:18091 couchbases:// protocol schemes. When using the couchbase:// or couchbases:// protocol schemes, you need not specify the port when connecting to the Couchbase cluster. -

    Command Line Option: -e or - --engine

    Example

    cbq> \CONNECT http://localhost:8093; +

    The cbq shell supports both IPV4 and IPV6 addresses.

    Command Line + Option: -e or + --engine

    Examples

    cbq> \CONNECT http://localhost:8093;cbq> \CONNECT http://[fd63:6f75:6368:1075:816:3c1d:789b:bc4]:8091     \DISCONNECT diff --git a/content/troubleshooting/troubleshooting-intro.dita b/content/troubleshooting/troubleshooting-intro.dita index 911304f2ee..76805df8e0 100644 --- a/content/troubleshooting/troubleshooting-intro.dita +++ b/content/troubleshooting/troubleshooting-intro.dita @@ -8,39 +8,12 @@ what you should do when you encounter them. You will also learn some general tips to keep in mind when troubleshooting your cluster, the various log files you might want to look into when troubleshooting an issue, and how you can report an issue to the Couchbase support.

    + +

    + For instructions on how to report issues to Couchbase, see + Contact Couchbase. +

    + -
    - Reporting Issues to Couchbase - -

    When reporting issues to Couchbase, add the following information to JIRA issues:

    -
      -
    • Provide a description of your environment (for example, package installation, - cluster_run, build number, operating system, and so on).
      • -
    • Show all the steps necessary to reproduce the issue (if applicable).
      • -
    • Show the full content of all the design documents.
      • -
    • Describe how your documents are structured (for example, if they are all same structure - or if they have different structures).
      • -
    • If you generated the data with a tool, identify the tool name and all the parameters - given to it (full command line).
      • -
    • Show the queries you were performing (include all query parameters and the full URL). - For example, if you are using curl, use the verbose option (-v) and - show the full output.

      The following request uses curl with the -v - option with test as the bucket, example as the design document, and Python (installed - separately) as the tool used to format the output. Replace node and bucket name in - example as appropriate: -

      curl -v http://couchbase:8092/test/_design/example | python -m json.tool
      • - -
    • Repeat the query with different values for the stale parameter and show the - output
      • -
    • Attach logs from all nodes in the cluster
      • -
    • Try all view related operations, including creating, updating, and deleting design - documents from the command line. The goal is to isolate UI problems from the view - engine.
      • -
    • If you suspect the indexer is stuck or blocked, use curl against the - _active_tasks API to isolate UI issues from view-engine issues. For example: - curl -s http://10.5.2.54:8092/_active_tasks -
      • -
    -
    diff --git a/content/understanding-couchbase.ditamap b/content/understanding-couchbase.ditamap new file mode 100644 index 0000000000..eaf4439e2d --- /dev/null +++ b/content/understanding-couchbase.ditamap @@ -0,0 +1,99 @@ + + + + + Understanding Couchbase + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/understanding-couchbase/buckets-memory-and-storage/bloom-filters.dita b/content/understanding-couchbase/buckets-memory-and-storage/bloom-filters.dita new file mode 100644 index 0000000000..d02877d020 --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/bloom-filters.dita @@ -0,0 +1,121 @@ + + + + Storage + When storing data in a Couchbase bucket, the server first writes data to the caching + layer and eventually stores all data to disk to provide a higher level of reliability. + +

    The Couchbase Server first writes data to the caching layer and puts the data into a disk write queue to be persisted to disk. Disk persistence enables you to perform backup and restore operations and to grow your datasets larger than the built-in caching layer. This disk storage process is called eventual persistence because the server does not block a client while it writes to disk.

    +

    If a node fails and all data in the caching layer is lost, the items can be recovered from disk. When the server identifies an item that needs to be loaded from disk because it is not in active memory, it places it in a load queue. A background process processes the load queue and reads the information back from disk and into memory. The client waits until the data is loaded back into memory before returning the information.

    +
    + +
    Multiple readers and writers
    +
    Multithreaded readers and writers provide simultaneous read and write operations for data on + disk. Simultaneous reads and writes increase I/O throughput. The multithreaded engine includes + additional synchronization among threads that are accessing the same data cache to avoid + conflicts. To maintain performance while avoiding conflicts over data, Couchbase Server uses a + form of locking between threads and thread allocation among vBuckets with static partitioning. +

    When Couchbase Server creates multiple reader and writer threads, the server assesses a + range of vBuckets for each thread and assigns each thread exclusively to certain vBuckets. + With this static thread coordination, the server schedules threads so that only a single + reader and single writer thread can access the same vBucket at any given time. The following + diagram shows six pre-allocated threads and two data buckets. Each thread has the range of + vBuckets that is statically partitioned for read and write access. + Bucket disk storage + +

    +     + +
    Item deletion
    +
    Items can be deleted explicitly by the client applications or deleted using an expiration flag. Couchbase Server never deletes items from disk unless one of these operations are performed. However, after deletion or expiration, a tombstone is maintained as the record of deletion. Tombstones help communicate the deletion or the expiration to downstream components. Once all downstream components have been notified, the tombstone gets purged as well.
    +     + +
    Tombstone purging
    +
    Tombstones are records of expired or deleted items that include item keys and metadata. +

    Couchbase Server and other distributed databases maintain tombstones in order to provide + eventual consistency between nodes and between clusters. Tombstones are records of expired or + deleted items and they include the key for the item and metadata. Couchbase Server stores the + key plus several bytes of metadata per deleted item in two structures per node. With millions + of mutations, the space taken up by tombstones can grow quickly. This is especially the case + if there are a large number of deletions or expired documents.

    The Metadata Purge + Interval sets frequency for a node to permanently purge metadata of deleted and expired items. + The Metadata Purge Interval setting runs as part of auto-compaction. This helps reduce the + storage requirement by roughly 3x times than before and also frees up space much faster.

     +
    + +
    Disk I/O priority +

    Disk I/O priority enables workload priorities to be set at the bucket level.

    +

    You can configure the bucket priority settings at the bucket level and set the value to be either high or low. Bucket priority settings determine whether I/O tasks for a bucket must be queued in the low or high priority task queues. Threads in the global pool poll the high priority task queues more often than the low priority task queues. When a bucket has a high priority, its I/O tasks are picked up at a higher frequency and thus, processed faster than the I/O tasks belonging to a low priority bucket.

    +

    You can configure the bucket I/O priority settings during initial setup and change the + settings later, if needed. However, changing a bucket I/O priority after the initial setup + results in a restart of the bucket and the client connections are reset. + Create bucket settings + +

    +

    The previous versions of Couchbase Server, version 3.0 or earlier, required the I/O thread + allocation per bucket to be configured manually. However, when you upgrade from a 2.x version to + a 3.x or higher version, Couchbase Server converts an existing thread value to either a high or + low priority based on the following criteria:

      +
    • Buckets allocated six to eight (6-8) threads in Couchbase Server 2.x are marked high + priority in bucket setting after the upgrade to 3.x or later.
      • +
    • Buckets allocated three to five (3-5) threads in Couchbase Server 2.x are marked low + priority in bucket settings after the upgrade to 3.x or later.
      • +

    + +
    +
    Monitoring Scheduler +

    You can use the cbstats command with the raw workload option to view the status of the threads as shown in the following example. + # cbstats 10.5.2.54:11210 -b default raw workload + + ep_workload:LowPrioQ_AuxIO:InQsize: 3 + ep_workload:LowPrioQ_AuxIO:OutQsize: 0 + ep_workload:LowPrioQ_NonIO:InQsize: 33 + ep_workload:LowPrioQ_NonIO:OutQsize: 0 + ep_workload:LowPrioQ_Reader:InQsize: 12 + ep_workload:LowPrioQ_Reader:OutQsize: 0 + ep_workload:LowPrioQ_Writer:InQsize: 15 + ep_workload:LowPrioQ_Writer:OutQsize: 0 + ep_workload:num_auxio: 1 + ep_workload:num_nonio: 1 + ep_workload:num_readers: 1 + ep_workload:num_shards: 4 + ep_workload:num_sleepers: 4 + ep_workload:num_writers: 1 + ep_workload:ready_tasks: 0 + ep_workload:shard0_locked: false + ep_workload:shard0_pendingTasks: 0 + ep_workload:shard1_locked: false + ep_workload:shard1_pendingTasks: 0 + ep_workload:shard2_locked: false + ep_workload:shard2_pendingTasks: 0 + ep_workload:shard3_locked: false + ep_workload:shard3_pendingTasks: 0

    +
    + +
    High Performance Storage +

    The scheduler and the shared thread pool provide high performance storage to the Couchbase Server.

    +
    + +
    Scheduler
    +
    The scheduler is responsible for managing a shared thread-pool and providing a fair + allocation of resources to the jobs waiting to execute in the vBucket engine. Shared thread + pool services requests across all buckets.

    As an administrator, you can govern the + allocation of resources by configuring a bucket’s disk I/O prioritization setting to be + either high or low.

     + +
    Shared thread pool
    +
    A shared thread pool is a collection of threads which are shared across multiple buckets for long running operations such as disk I/O. Each node in the cluster has a thread pool that is shared across multiple vBuckets on the node. Based on the number of CPU cores on a node, the database engine spawns and allocates threads when a node instance starts up. +

    Using a shared thread pool provides the following benefits:

      +
    • Better parallelism for worker threads with more efficient I/O resource management.
      • +
    • Better system scalability with more buckets being serviced with fewer worker threads.
      • +
    • Availability of task priority if the disk bucket I/O priority setting is used.
      • +

    +
    +
    + + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/buckets-memory-and-storage.dita b/content/understanding-couchbase/buckets-memory-and-storage/buckets-memory-and-storage.dita new file mode 100644 index 0000000000..cae13e29f6 --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/buckets-memory-and-storage.dita @@ -0,0 +1,56 @@ + + + + + + Buckets, Memory, and Storage + + + + Couchbase Server saves items in Buckets. + + + + +
    + + + Creating, Maintaining, and Accessing Data + + +

    + Couchbase Server keeps items in Buckets. Before an item can be saved, + a bucket must exist for it. Buckets + can be created by means of the Couchbase Web Console, the REST API, or the + CLI. Each bucket is assigned a name at its creation: this name is referenced by the + application or user wishing to save or access items within it. +

    + +

    + The types of bucket most frequently used are Couchbase and + Ephemeral buckets. Couchbase buckets exist both in memory and on disk. + Ephemeral buckets exist only in memory. Buckets can be configured to maintain data + in compressed format, so as to maximize resource-effectiveness. Documents + within a bucket can be assigned a Time To Live, to ensure that they expire + after a specified time-period, and that their data is thus made permanently inaccessible. +

    + +

    + Couchbase Server manages memory to ensure high performance and scalability: memory + quotas are established, and data not recently used can be ejected, to make room for + data more frequently requested. Multi-threaded readers and writers provide simultaneous + read and write operations for data on disk, ensuring high throughput. +

    + +

    + The pages in this section provide detailed descriptions of buckets, memory, and storage. +

    + + +
    + + + + + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/buckets.dita b/content/understanding-couchbase/buckets-memory-and-storage/buckets.dita new file mode 100644 index 0000000000..bb8ce9d739 --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/buckets.dita @@ -0,0 +1,503 @@ + + + + + Buckets + + + + Couchbase Server uses buckets to group collections of keys and values logically. + + + + +
    + + + Bucket Types + + +

    + Three types of bucket are provided: +

    + +
      +
    • + Couchbase buckets: These store data persistently, as well as in memory. + They allow data to be automatically replicated for high availability, using the + Database + Change Protocol (DCP); and dynamically scaled across multiple clusters, by means of + Cross Datacenter Replication (XDCR). + +

      + +

      + +

      + If a Couchbase bucket's RAM-quota is exceeded, items are ejected. This means + that data, which is resident both in memory and on disk, is removed from memory, but + not from disk. Therefore, if removed data is subsequently needed, it is reloaded + into memory from disk. For a Couchbase bucket, ejection can be either of the following, + based on + configuration performed at the time of bucket-creation: + +

        + +
      • + Value-only: Only key-values are removed. Generally, this favors performance + at the expense of memory. + +

        + + +

        + +
        • + +
      • + Full: All data — including keys, key-values, and metadata — is removed. Generally, + this favors memory at the expense of performance. + +
        • + +
      + +

      + +
      • + +
    • + Ephemeral buckets: These are an alternative to Couchbase buckets, to be used whenever persistence + is not required: for example, when repeated disk-access involves too much overhead. This allows + highly consistent in-memory performance, without disk-based + fluctuations. It also allows faster node rebalances and restarts. + +

      + +

      + +

      + If an Ephemeral bucket's RAM-quota is exceeded, one of the following occurs, based on configuration + performed at the time of + bucket-creation: +

      + +
        + +
      • + Resident data-items remain in RAM. No additional data can be added; and attempts to add data therefore fail. + +

        + +

        + +
        • + +
      • + Resident data-items are ejected from RAM, to make way for new data. For + an Ephemeral bucket, this means + that data, which is resident in memory (but, due to + this type of bucket, can never be on disk), is + removed from memory. Therefore, if removed + data is subsequently needed, it cannot be re-acquired from Couchbase Server. + +

        + +

        + +

        + For an Ephemeral bucket, ejection removes all of an item's data: however, + a tombstone (a record of the ejected item, which + includes keys and metadata) is retained until the next scheduled purge + of metadata for the current node. See + Storage + for more information. +

        + +

        + +

        + +
        • + +
      + +
      • + +
    • + Memcached buckets: These are designed to be used alongside other database platforms, such as ones + employing relational database technology. By caching frequently-used data, Memcached buckets reduce the number + of queries a database-server must perform. Each + Memcached bucket provides a directly addressable, distributed, in-memory key-value cache. + +

      + +

      + +

      + Memcached buckets are not persistent on disk: they only exist in RAM. If a Memcached bucket's + RAM-quota is exceeded, items are ejected. For a Memcached bucket, this means that data, + which is resident in memory (but, due to + this type of bucket, can never be on disk), is + removed from memory. Therefore, if removed + data is subsequently needed, it cannot be re-acquired from Couchbase Server. Ejection + removes all of an item's data. +

      + +

      + +

      + +
      • + +
    + +

    + For all bucket-types, items are selected for ejection by means of the + Not Recently Used (NRU) algorithm. +

    + +

    + All bucket types are fully compatible with the Memcached open source distributed key-value cache. +

    + +
    + +
    + + + Bucket Capabilities + + +

    + Each bucket type supports a different set of capabilities; as shown by the following + table. +

    + +
    + + Bucket types and associated capabilities + + + + + + + + + + Capability + Memcached buckets + Couchbase buckets + Ephemeral buckets + + + + + + + Item size limit + 1 MB + 20 MB + 20 MB + + + + Persistence + No + Yes + No + + + + Replication (DCP) + No + Yes + Yes + + + + Cross Datacenter Replication (XDCR) + No + Yes + Yes + + + + Rebalance + No + Yes + Yes + + + + Statistics + All except disk-related + All + All except disk-related + + + + Client support + Ketama consistent hashing + Full smart client support + Full smart client support + + + + Backup + No + Yes + Yes + + + + Querying via N1QL + No + Yes + Yes + + + + Map-reduce & spatial views + No + Yes + No + + + + Standard GSI Storage + No + Yes + No + + + + Memory Optimized Index Storage + No + Yes + Yes + + + + Full-Text Search + No + Yes + Yes + + + + External connectors + No + Yes + Yes + + + + Encrypted data access + Yes + Yes + Yes + + + + Durability: Persist To + No + Yes + No + + + + Durability: Replication + No + Yes + Yes + + + + TTL + Yes + Yes + Yes + + + + Bucket TTL + No + Yes + Yes + + + + Data Compression + No + Yes + Yes + + + + +
    + +
    + +
    + + Couchbase vs Ephemeral Buckets + + +

    + Couchbase and Ephemeral buckets both provide a highly available, dynamically reconfigurable, distributed + data-store; which survives node failures, and allows cluster-reconfiguration to occur while service-requests + continue to be handled. The following + table compares the principal capabilities shared by both bucket-types. +

    + + + + + Couchbase and Ephemeral bucket-capabilities compared + + + + + + + + + Capability + Couchbase Buckets + Ephemeral Buckets + + + + + + + + + Persistence + + + + Couchbase buckets are persisted asynchronously, from memory to disk. This + provides protection from server-restarts. You can set persistence-properties at the + bucket level. + + + + Ephemeral buckets are not persisted to disk: they are retained in RAM only. + + + + + + + + Replication (DCP and XDCR) + + + + Couchbase buckets can be replicated across a configurable number of servers. + If the host machine fails, a replica server is promoted to be the host server, + providing high availability cluster operations via failover. You can configure replication + at the bucket level. Additionally, Cross Datacenter Replication (XDCR) allows + replication of Couchbase bucket-data between clusters. + + + + Ephemeral buckets can be replicated across a configurable number of servers, + exactly as can Couchbase buckets; but without being persisted to disk. + Likewise, XDCR allows replication of + Ephemeral bucket-data between clusters and without persistence: note however, that + an Ephemeral bucket configured to eject data when its RAM-quota is exceeded + is not permitted to be used as + a source for XDCR (though it is permitted to be used as a target). + + + + + + + + Rebalance + + + + By means of rebalancing, the load constituted by Couchbase buckets is distributed + evenly across nodes within the cluster. Buckets and nodes can be dynamically added and + removed. + + + + Rebalancing redistributes Ephemeral buckets, exactly as it does Couchbase buckets; but + without the data being persisted to disk. + + + + + + + + Auto-failover + + + + By default, Auto-failover starts when a node has been inaccessible for 120 seconds. Auto-failover + can happen only once, for a single node; prior to manual reset, performed by an + administrator. When a failed node becomes accessible again, delta-node recovery is used: + re-using data on disk, and resynchronizing it. + + + + Auto-failover starts as soon as a node is inaccessible. + Auto-failover can happen multiple times, for multiple nodes. When a failed node becomes + accessible again, no delta-node recovery is required, since no data resides on disk. + + + + + + +
    + + +
    + +
    + + + Bucket Security + + +

    + Buckets are protected by the Couchbase Role-Based Access Control + (RBAC) system. See + Authorization + and + Authentication + for details. +

    + +

    + Legacy buckets, defined on releases of Couchbase Server prior to 5.0, may continue + to be accessible without password-specification. However, you are strongly recommended + to ensure that all buckets are fully protected by RBAC, especially for production + purposes. +

    + +
    + +
    + + Using Buckets in Administration and Development + + +

    + For information on how to create, access, and manage buckets, see + Setting Up Buckets. +

    +
    + + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/compression.dita b/content/understanding-couchbase/buckets-memory-and-storage/compression.dita new file mode 100644 index 0000000000..2019b0e0db --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/compression.dita @@ -0,0 +1,273 @@ + + + + + + + + Compression + + + + Couchbase Server supports data compression in its communications + with internal and external clients, and in its internal handling of items. + + + + +
    + + + Overview + + +

    + The compression of data allows RAM and disk-space to be used with increased efficiency. It may + also reduce consumption of network bandwidth. Higher consumption of CPU resources may result. +

    + +

    + Compression, provided by the open-source library + Snappy, + is applied to items based both on the client's capabilities and on the compression mode established + by the user for the given bucket. +

    + +

    + Compression is available only in Couchbase Enterprise Edition, and can be applied only to Couchbase and + Ephemeral buckets. Compression applies to both binary and JSON items. +

    + +
    + +
    + + + Where Compression is Used + + +

    + +

    + +

    + Clients based on the Couchbase SDK (1), nodes within a cluster that participate + in intra-cluster replication (4), internal Couchbase Services (5), + external DCP clients (6), and remote clusters that participate in Cross Data-Center + Replication (7) communicate their ability to send and receive compressed + items by using the HELO command, with a flag that confirms support of the Snappy + Compression data type. +

    + +

    + Couchbase Server may (depending on the mode of the bucket) store items in compressed form in + memory (2). The server always compresses + items when storing them on disk (3). +

    + +
    + +
    + + + Compression Modes + + +

    + Each bucket + is configured to support one of three modes. After a client has communicated + its ability to send and receive compressed data, the server's running of compression and decompression routines + depends on the + mode supported by the specified bucket. The modes are as follows: +

    + +

    + +

    + +
      + +
    • + + Off: Provides the + behavior of Couchbase Server pre-5.5. On receipt of a compressed item, + Couchbase Server decompresses the item when storing in memory; and recompresses it + when storing on disk. Couchbase Server + sends the item in uncompressed form. + This mode is assigned by default to buckets upgraded from a previous + version of Couchbase Server. + The mode is recommended for use where clients cannot benefit from compression, and + where neither memory-resources nor network-bandwidth will be negatively impacted by the size and + quantity of items to be handled. Note also that this is the only mode under which Memcached buckets + operate. + + +
      • + +
    • + Passive: On receipt of a compressed item, + Couchbase Server stores it in compressed form both in memory + and on disk. Couchbase Server + sends the item back to the client in compressed form if this is requested by the client; otherwise, + it sends the item back in uncompressed form. + On receipt of an uncompressed item, Couchbase Server + stores it in memory in uncompressed form, and stores it on disk in compressed form. + It returns the item to the client in uncompressed form. + This mode is assigned by default to new buckets, in + Couchbase Server 5.5 and beyond. It supports clients that themselves handle compresion, and + additionally allows Couchbase Server to + to limit its use of memory-resources and network-bandwidth. At the same time, it does not force + Couchbase Server to use CPU + resources for the compression and decompression of items that clients do not + themselves require in compressed form. + + +
      • + +
    • + Active: Couchbase Server actively compresses items for storage in memory + and on disk, even if the items are received in uncompressed form. + Items are decompressed before being sent back to those clients that do not + support the receiving of compressed data. Items are sent + in compressed form to clients that do support the receiving of compressed data, even if those + clients originally sent the + items to the server in uncompressed form. + This mode allows the server to practice the maximum conservation of memory-resources + and network-bandwith. Consequently, more items can be held in memory simultaneously, and thereby + accessed with improved, overall efficiency (since the processing-time required for compression and + decompression is significantly less than that required for fetching data from disk). Nevertheless, in some circumstances, + clients that do not themselves require the compression and decompression of items may be negatively affected by + the compression and decompression performed by the server on items resident in memory. +
      • + +
    + +
    + + +
    + + + Switching Between Compression Modes + + +

    + Buckets can be switched between modes. The following behavior-changes + should be noted: +

    + +
      + +
    • + When a bucket formerly in Passive mode has been switched to Off mode, + any compressed item that Couchbase Server receives for that bucket + is stored in memory in uncompressed form. If the server + needs to send from the bucket an item that is currently compressed, the server decompresses + the item before sending: however, to preserve + memory-efficiency, the + item remains compressed in memory. + +

      + +

      + +
      • + +
    • + When a bucket has been switched from Passive to Active mode, it + periodically runs a task that compresses uncompressed items. + +

      + +

      + +
      • + +
    • + When a bucket formerly in Active mode is switched to Off mode, + this disables + the task that was periodically run to compress uncompressed items. + Compressed items can continue to be received for the bucket, and are + decompressed for storage in memory. Compressed items within the bucket are + decompressed before sending: however, to ensure continued + memory-efficiency, the + item remains compressed in memory. + +

      + +

      + +
      • + +
    + +
    + + + +
    + + + Enabling Compression + + +

    + Couchbase Server allows authorized users to configure + compression by means of the following. +

    + +
      + +
    • +

      + The Couchbase Web Console, + the Couchbase CLI, and the Couchbase REST API, which allow buckets each to be assigned a compression mode; and which allow + Cross Data-Center Replication (XDCR) to handle compressed items. +

      + +

      + For information on using the Couchbase Web Console to create buckets, see + Create a Bucket; and + for information on using it to set up XDCR, see + Managing XDCR +

      + +

      + +

      +
      • + + +
    • +

      + The cbbackup, cbrestore, and + associated tools, which allow + compressed items to be requested in the backup stream. +

      +

      + +

      +
      • + +
    • +

      + The Couchbase SDK, which can elect to send and receive compressed items. +

      +

      + +

      +
      • + +
    + +

    + For information on roles that allow modification of bucket-settings, see + Authorization. +

    + +
    + + + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/db-engine-architecture.dita b/content/understanding-couchbase/buckets-memory-and-storage/db-engine-architecture.dita new file mode 100644 index 0000000000..35dcaf4113 --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/db-engine-architecture.dita @@ -0,0 +1,293 @@ + + + + Database Engine Architecture + The memory-first architecture of the Couchbase Server enables it to maintain sub-millisecond latencies with core data access. + +

    The Couchbase Server depends on the following key components:

      +
    • A highly efficient listener that manages networking and authentication.
      • +
    • A bucket engine that stores and retrieves information at the speed of memory access. +
      • +
    With Couchbase buckets, data is stored on disk eventually through the storage engine. The + storage engine enables the server to efficiently hold data much larger than the size of memory. + + Database engine architecture + +

    +
    Listeners +

    When client connection requests arrive at the database engine, the listener service receives the requests and authenticates the client. Upon successful authentication, the listener service assigns a worker thread to the connection to service its request. A single worker thread can handle multiple client connections using a non-blocking event loop. +

    +

    The number of worker threads that can be created is automatically determined based on the number of CPU threads present on the node. By default the number of worker threads is 0.75 x number of CPU threads.

    +
    vBucket manager and managed cache +

    After executing mutation and read requests, the server uses the managed cache to hold updated and newly created values. However, with a high flow of incoming operations, the system can run out of memory quickly. In order to reuse the memory, mutations are also queued for disk persistence. Once the mutated items are persisted, the server frees up the memory consumed by these items, making space for newer operations. This operation is called cache eviction. With a highly concurrent set of operations consuming memory and a high throughput disk subsystem persisting data to disk, there can be many pages eligible for reuse. The server uses the Least Recently Used (LRU) algorithm to identify the memory pages that can be reused.

    +

    It is important to size the RAM capacity appropriately for your working set: the portion of data that your application is working with at any given point in time and needs very low latency and high throughput access. In some applications, the working set is the entire data set, while in others it is a smaller subset.

    +
    +
    Initialization and Warmup +

    Whenever you restart the Couchbase Server or restore the data, the node goes through a warmup process before it starts handling data requests again. During warmup, the Couchbase Server loads data persisted on disk into RAM.

    +

    Couchbase Server provides an optimized warmup process that loads data sequentially from disk + into RAM. It divides the data to be loaded and handles it in multiple phases. After the warmup + process completes, the data is available for clients to read and write. The time needed for a + node warmup depends on the system size, system configuration, the amount of data persisted in + the node, and the ejection policy configured for the buckets.

    +

    Couchbase Server identifies items that are frequently used, prioritizes them, and loads them + before sequentially loading the remaining data. The frequently-used items are prioritized in an + access log. The server performs a prefetch to get a list of the most frequently accessed keys + and then fetches these keys before fetching any other items from disk.

    +

    The server runs a configurable scanner process that determines the keys that are most frequently used. The scanner process is preset and is configurable. You can use the command-line tool,cbepctl flush_param, to change the initial time and interval for the scanner process. For example, you can configure the scanner process to run during a specific time period when a given list of keys need to be identified and made available sooner.

    +

    The server can also switch into a ready mode before it has actually retrieved all documents for keys into RAM, thereby enabling data to be served before all the stored items are loaded. Switching into ready mode is a configurable setting that enables you to adjust the server warmup time.

    +
    +
    Tunable Memory with Ejection Policy +

    Tunable memory enables you to configure the ejection policy for a bucket as one of the following:

      +
    • Value-only ejection (default) removes data from the cache but keeps all keys + and metadata fields for non-resident items. When a value bucket ejection occurs, the value of + the item is reset. Value-only ejection, also referred to as value ejection, is well suited for + cases where low latency access is critical to the application and the total item keys for the + bucket can easily fit in the allocated Data RAM quota.
      • +
    • Full metadata ejection removes all data including keys, metadata, and + key-value pairs from the cache for non-resident items. Full ejection is well suited for cases + where the application has cold data that is not accessed frequently or the total data size is + too large to fit in memory plus higher latency access to the data is accepted. The performance + of full eviction cache management is significantly improved by Bloom filters. Bloom filters are enabled by default and cannot be disabled. Full ejection may involve additional disk I/O per operation. For + example, when the request get_miss which requests a key that does not exist is + received, Couchbase Server will check for the key on the disk even if the bucket is 100% + resident.
      • +

    +
    +
    Working Set Management and Ejection +

    Couchbase Server actively manages the data stored in a caching layer; this includes the information which is frequently accessed by clients and which needs to be available for rapid reads and writes. When there are too many items in RAM, Couchbase Server removes certain data to create free space and to maintain system performance. This process is called "working set management" and the set of data in RAM is referred to as the "working set". In general, the working set consists of all the keys, metadata, and associated documents which are frequently used require fast access. The process the server performs to remove data from RAM is known as ejection.

    +

    Couchbase Server performs ejections automatically. When ejecting information, it works in conjunction with the disk persistence system to ensure that data in RAM is persisted to disk and can be safely retrieved back into RAM whenever the item is requested.

    +

    In addition to the Data RAM quota for the caching layer, the engine uses two watermarks, + mem_low_wat and mem_high_wat, to determine when it + needs to start persisting more data to disk.

    +

    As more and more data is held in the caching layer, at some point in time it passes the + mem_low_wat value. At this point, no action is taken. As data continues to + load, it eventually reaches the mem_high_wat value. At this point, the + Couchbase Server schedules a background job called item pager which ensures that items are + migrated to disk and memory is freed up for other Couchbase Server items. This job runs until + measured memory reaches mem_low_wat. If the rate of incoming items is + faster than the migration of items to disk, the system returns errors indicating there is not + enough space until there is sufficient memory available. The process of migrating data from the + cache to make way for actively used information is called ejection and is controlled + automatically through thresholds set on each configured bucket in the Couchbase Server cluster. + + Working set management and ejection + +

    +

    Depending on the ejection policy set for the bucket, the vBucket Manager removes just the + document or both the document, key and the metadata for the item being ejected. Keeping an + active working set with keys and metadata in RAM serves three important purposes in a system:

      +
    • Couchbase Server uses the remaining key and metadata in RAM if a client requests for that + key. Otherwise, the node tries to fetch the item from disk and return it into RAM.
      • +
    • The node can also use the keys and metadata in RAM for miss access. This means that + it can quickly determine whether an item is missing and if so, perform some action, such as + add it.
      • +
    • The expiration process in Couchbase Server uses the metadata in RAM to quickly scan for + items that have expired and later removes them from disk. This process is known as expiry + pager and runs every 60 minutes by default.
      • +

    +
    +
    Not Recently Used (NRU) Items +

    All items in the server contain metadata indicating whether the item has been recently accessed or not. This metadata is known as not-recently-used (NRU). If an item has not been recently used, then the item is a candidate for ejection. When data in the cache exceeds the high water mark (mem_high_wat), the server evicts items from RAM.

    +

    Couchbase Server provides two NRU bits per item and also provides a replication protocol that can propagate items that are frequently read, but not mutated often.

    +

    NRUs are decremented or incremented by server processes to indicate an item that is more + frequently or less frequently used. The following table lists the bit values with the + corresponding scores and statuses: + Scoring for NRU bit values + + + + + + + + Binary NRU + Score + Access pattern + Description + + + + + 00 + 0 + Set by write access to 00. Decremented by read access or no access. + Most heavily used item. + + + 01 + 1 + Decremented by read access. + Frequently accessed item. + + + 10 + 2 + Initial value or decremented by read access. + Default value for new items. + + + 11 + 3 + Incremented by item pager for eviction. + Less frequently used item. + + + +

    +

    There are two processes that change the NRU for an item: +

      +
    • When a client reads or writes an item, the server decrements NRU and lowers the item's score.
      • +
    • A daily process which creates a list of frequently-used items in RAM. After the completion of this process, the server increments one of the NRU bits.
      • +
    + Because these two processes change NRUs, they play an important role in identifying the candidate items for ejection.

    +

    You can configure the Couchbase Server settings to change the behavior during ejection. For example, you can specify the percentage of RAM to be consumed before items are ejected, or specify whether ejection should occur more frequently on replicated data than on original data. Couchbase recommends that the default settings be used.

    +
    +
    Understanding the Item Pager +

    The item pager process runs periodically to remove documents from RAM. When the amount of RAM + used by items reaches the high water mark (upper threshold), both active and replica data are + ejected until the amount of RAM consumed (memory usage) reaches the low water mark (lower + threshold). Using the default settings, active documents have a 40% chance of eviction while + replica documents have a 60% chance of eviction. Both the high water mark and low water mark are + expressed as a percentage amount of RAM, such as 80%.

    +

    You can change the high water mark and low water mark settings for a node by specifying a + percentage amount of RAM, for example, 80%. Couchbase recommends that you use the following + default settings: + Default setting for RAM water marks + + + + + + + Version + High water mark + Low water mark + + + + + 2.0 + 75% + 60% + + + 2.0.1 and higher + 85% + 75% + + + +

    +

    The item pager ejects items from RAM in two phases: +

      +
    1. Eject items based on NRU: The item pager scans NRU for items, creates a list of items with a NRU score 3, and ejects all the identified items. It then checks the RAM usage and repeats the process if the usage is still above the low water mark.
      2. +
    2. Eject items based on algorithm: The item pager increments the NRU of all items by 1. For + every item whose NRU is equal to 3, it generates a random number. If the random number for an + item is greater than a specified probability, it ejects the item from RAM. The probability is + based on the current memory usage, low water mark, and whether a vBucket is in an active or + replica state. If a vBucket is in an active state, the probability of ejection is lower than + if the vBucket is in a replica state. + Probability of ejection based on active vBuckets versus replica vBuckets (using + default settings) + + + + + + Active vBucket + Replica vBucket + + + + + 40% + 60% + + + +
      3. +

    +
    +
    Active Memory Defragmenter +

    Over time, the memory used by the managed cache of a running Couchbase Server can become fragmented. The storage engine now includes an Active Defragmenter task to defragment cache memory.

    +

    Cache fragmentation is a side-effect of how Couchbase Server organizes cache memory to maximize performance. Each page in the cache is typically responsible for holding documents of a specific size range. Over time, if memory pages assigned to a specific size range become sparsely populated (due to documents of that size being ejected or items changing in size), then the unused space in those pages cannot be used for documents of other sizes until a complete page is free and that page is re-assigned to a new size. Such effects are highly workload dependent and can result in memory that cannot be used efficiently by the managed cache.

    +

    The Active Memory Defragmenter attempts to address any fragmentation by periodically scanning the cache to identify pages which are sparsely used, and repacking the items stored on those pages to free up whole pages.

    +
    +
    High Performance Storage +

    The scheduler and the shared thread pool provide high performance storage to the Couchbase Server.

    +
    + +
    Scheduler
    +
    The scheduler is responsible for managing a shared thread-pool and providing a fair + allocation of resources to the jobs waiting to execute in the vBucket engine. Shared thread + pool services requests across all buckets.

    As an administrator, you can govern the + allocation of resources by configuring a bucket’s disk I/O prioritization setting to be + either high or low.

     + +
    Shared thread pool
    +
    A shared thread pool is a collection of threads which are shared across multiple buckets for long running operations such as disk I/O. Each node in the cluster has a thread pool that is shared across multiple vBuckets on the node. Based on the number of CPU cores on a node, the database engine spawns and allocates threads when a node instance starts up. +

    Using a shared thread pool provides the following benefits:

      +
    • Better parallelism for worker threads with more efficient I/O resource management.
      • +
    • Better system scalability with more buckets being serviced with fewer worker threads.
      • +
    • Availability of task priority if the disk bucket I/O priority setting is used.
      • +

    +
    +
    +
    Disk I/O priority +

    Disk I/O priority enables workload priorities to be set at the bucket level.

    +

    You can configure the bucket priority settings at the bucket level and set the value to be either high or low. Bucket priority settings determine whether I/O tasks for a bucket must be queued in the low or high priority task queues. Threads in the global pool poll the high priority task queues more often than the low priority task queues. When a bucket has a high priority, its I/O tasks are picked up at a higher frequency and thus, processed faster than the I/O tasks belonging to a low priority bucket.

    +

    You can configure the bucket I/O priority settings during initial setup and change the + settings later, if needed. However, changing a bucket I/O priority after the initial setup + results in a restart of the bucket and the client connections are reset. + Create bucket settings + +

    +

    The previous versions of Couchbase Server, version 3.0 or earlier, required the I/O thread + allocation per bucket to be configured manually. However, when you upgrade from a 2.x version to + a 3.x or higher version, Couchbase Server converts an existing thread value to either a high or + low priority based on the following criteria:

      +
    • Buckets allocated six to eight (6-8) threads in Couchbase Server 2.x are marked high + priority in bucket setting after the upgrade to 3.x or later.
      • +
    • Buckets allocated three to five (3-5) threads in Couchbase Server 2.x are marked low + priority in bucket settings after the upgrade to 3.x or later.
      • +

    + +
    +
    Monitoring Scheduler +

    You can use the cbstats command with the raw workload option to view the status of the threads as shown in the following example. + # cbstats 10.5.2.54:11210 -b default raw workload + + ep_workload:LowPrioQ_AuxIO:InQsize: 3 + ep_workload:LowPrioQ_AuxIO:OutQsize: 0 + ep_workload:LowPrioQ_NonIO:InQsize: 33 + ep_workload:LowPrioQ_NonIO:OutQsize: 0 + ep_workload:LowPrioQ_Reader:InQsize: 12 + ep_workload:LowPrioQ_Reader:OutQsize: 0 + ep_workload:LowPrioQ_Writer:InQsize: 15 + ep_workload:LowPrioQ_Writer:OutQsize: 0 + ep_workload:num_auxio: 1 + ep_workload:num_nonio: 1 + ep_workload:num_readers: 1 + ep_workload:num_shards: 4 + ep_workload:num_sleepers: 4 + ep_workload:num_writers: 1 + ep_workload:ready_tasks: 0 + ep_workload:shard0_locked: false + ep_workload:shard0_pendingTasks: 0 + ep_workload:shard1_locked: false + ep_workload:shard1_pendingTasks: 0 + ep_workload:shard2_locked: false + ep_workload:shard2_pendingTasks: 0 + ep_workload:shard3_locked: false + ep_workload:shard3_pendingTasks: 0

    +
    + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/expiration.dita b/content/understanding-couchbase/buckets-memory-and-storage/expiration.dita new file mode 100644 index 0000000000..1af1e40ec0 --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/expiration.dita @@ -0,0 +1,321 @@ + + + + + + + + Expiration + + + + Bucket Time To Live (TTL) imposes a maximum lifespan on items within a bucket, and thus + ensures the expiration of such items, once the specified period is complete. + + + + +
    + + + Overview + + +

    + Bucket TTL is a non-negative integer-value, specified per bucket, that determines the maximum expiration times + of individual items: if Bucket TTL is enabled, each newly created + item lives for the number of seconds specified by Bucket TTL, following the item's creation. After its + expiration time is reached, the item + will be deleted by Couchbase Server. Deleted items and their contents are no longer available: + access-attempts are treated + as if the items never existed. +

    + +

    + The maximum value for Bucket TTL is MAX32INT seconds (2147483648, or 68.096 years). The default value is 0, which + indicates that Bucket TTL is disabled. If Bucket TTL is changed from the default, it is thereby enabled. +

    + +

    + Whenever Bucket TTL is modified, items that existed prior to the modification: +

    + +
      +
    • + Remain subject to the previous Bucket TTL, if Bucket TTL was previously enabled. + +

      + +

      + +
      • + +
    • + Remain subject to no Bucket TTL, if Bucket TTL was not previously enabled. + +

      + +

      + +
      • + +
    + +

    + Items created or modified following the modification of Bucket TTL are subject to the modified Bucket TTL. +

    + +

    + Bucket TTL can be established on Couchbase and Ephemeral buckets. It cannot be established on + Memcached buckets. Note that Bucket TTL is only available in the Enterprise Edition of Couchbase Server. +

    + + +
    + +
    + + + Expiration: Bucket versus Item + + +

    + As described in the Expiration Overview section of + Core Operations, + an expiration time can be specified per item, by means of Couchbase SDK APIs: this is referred + to as the item’s TTL. If a new item’s TTL is specified to be: + +

    + +
      +
    • + Above that of Bucket TTL, the item's TTL is reduced to the value of Bucket TTL. + +

      + +

      + +
      • + +
    • + Below that of Bucket TTL, the item's TTL is left unchanged. + +

      + +

      + +
      • + +
    • + 0, the item's TTL is reset to the value of Bucket TTL. + +

      + +

      + +
      • + +
    + +
    + +
    + + + Bucket TTL Use Cases + + +

    + Bucket TTL is intended for use with buckets containing data that either must or can be permitted to expire after + a certain period of time. +

    + +

    + Bucket TTL must not be used with buckets whose data is required not to expire: for example, the buckets + that support Couchbase Eventing and Couchbase Mobile. If Bucket TTL is used with such buckets, system failures + may result. +

    + +
    + +
    + + + Post-Expiration Purging + + +

    + When its expiration time is reached, an item is deleted + as soon as one of the following occurs: +

    + +
      +
    • + An attempt is made to access the item. +

      + +

      +
      • + +
    • + The expiry pager is run. For information on the expiry pager, see + Memory. +

      + +

      +
      • + +
    • + Compaction is run. For information on performing compaction with the Couchbase CLI, see + bucket-compact; + with Couchbase REST APIs, see + Compaction API; with + the Couchbase Web Console (as auto-compaction), see + Configuring Auto-Compaction. + +

      + +

      +
      • +
    + +

    + As described in + Bucket Disk Storage, + for each item that is deleted, a tombstone, which includes key and metadata, continues to be maintained by + Couchbase Server for some period of time, as a record: this applies to all instances of item-deletion, + including those achieved by specifying Bucket TTL. To ensure that no trace of deleted items remains, + tombstones are + removed through a Metadata Purge. + This is an automatic, non-disruptive background-process, which can be scheduled if necessary by means of + Couchbase Web Console; as described in + Configuring Auto-Compaction. + Note that the console allows the intervals between purges to be established as appropriately low, so + that tombstones are removed promptly. +

    + +
    + +
    + + + Bucket-Expiration and XDCR + + +

    + When Cross Data-Center Replication occurs, the Bucket TTL setting does not get propagated from the source bucket + to the target. However, items within the source bucket that have individual expiration times, including ones + derived from the source Bucket TTL setting, are replicated to the target along with their individual expiration times. +

    + +

    + Buckets in the target cluster can have their own Bucket TTL settings, which may differ from those + of the buckets in the source. If, on the target cluster, the TTL of a replicated item is: +

    + +
      +
    • + Above that of Bucket TTL, the item's TTL is reduced to the value of Bucket TTL. + +

      + +

      +
      • + +
    • + Below that of Bucket TTL, the item's TTL is left unchanged. + +

      + +

      + +
      • + +
    • + 0, the item's TTL is reset to the value of Bucket TTL. + +

      +

      +
      • +
    + +

    + Note that in cases where the TTL of the replicated item is above that of Bucket TTL or is 0, if bi-directional + XDCR has been set up, the TTL of the item at the source is + eventually also either reduced or reset to the value of the target's Bucket TTL. +

    + +

    + Note also that when an item is replicated by XDCR, its expiration time is communicated to the target as absolute. + This requires that the system clocks of the respective clusters be fully synchronized; otherwise, inconsistent + behavior may result (as, for example, in the case where an item arrives at the target-cluster with an absolute + expiration time that is earlier than the current time acknowledged by the target-cluster). See + Clock Sync with NTP. +

    + +
    + +
    + + + Setting Bucket-Expiration + + +

    + An expiration time can be set for a bucket by means of any of the following: +

    + +
      + +
    • + The UI provided by Couchbase Web Console. See + Create a Bucket. +

      +

      + +
      • + +
    • + The Couchbase CLI. See + bucket-create. + +

      +

      + +
      • + +
    • + The Couchbase REST API. See + Creating and Editing Buckets. +

      +

      + +
      • + +
    + +

    + For information on roles that allow modification of bucket-settings, see + Authorization. +

    + +
    + +
    + + + Auditing + + +

    + If auditing is switched on, changes to each bucket's expiration time are + recorded, and can be subsequently viewed. See + Auditing. +

    + +
    + + + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/bucket-disk-storage.png b/content/understanding-couchbase/buckets-memory-and-storage/images/bucket-disk-storage.png new file mode 100644 index 0000000000..ef6d02dfca Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/bucket-disk-storage.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/compression-overview-diagram.png b/content/understanding-couchbase/buckets-memory-and-storage/images/compression-overview-diagram.png new file mode 100644 index 0000000000..9c072d966c Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/compression-overview-diagram.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/compressionDiagram.png b/content/understanding-couchbase/buckets-memory-and-storage/images/compressionDiagram.png new file mode 100644 index 0000000000..638682f725 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/compressionDiagram.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/create-bucket-settings.png b/content/understanding-couchbase/buckets-memory-and-storage/images/create-bucket-settings.png new file mode 100644 index 0000000000..b3c72e8a9e Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/create-bucket-settings.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence1.png b/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence1.png new file mode 100644 index 0000000000..a066074b25 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence1.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence2.png b/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence2.png new file mode 100644 index 0000000000..babfbdee15 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence2.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence3.png b/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence3.png new file mode 100644 index 0000000000..7513a3f25d Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence3.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence4.png b/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence4.png new file mode 100644 index 0000000000..01d4ec9d42 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/createDocSequence4.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/db-engine-architecture.png b/content/understanding-couchbase/buckets-memory-and-storage/images/db-engine-architecture.png new file mode 100644 index 0000000000..a83fd53d1e Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/db-engine-architecture.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/tunable-memory-bucket-config.png b/content/understanding-couchbase/buckets-memory-and-storage/images/tunable-memory-bucket-config.png new file mode 100644 index 0000000000..660f24e84d Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/tunable-memory-bucket-config.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/tunableMemory.png b/content/understanding-couchbase/buckets-memory-and-storage/images/tunableMemory.png new file mode 100644 index 0000000000..7869c030a8 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/tunableMemory.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence1.png b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence1.png new file mode 100644 index 0000000000..2c70f12d80 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence1.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence2.png b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence2.png new file mode 100644 index 0000000000..351149cd64 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence2.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence3.png b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence3.png new file mode 100644 index 0000000000..b7ea88ca03 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence3.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence4.png b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence4.png new file mode 100644 index 0000000000..04c70fbe4e Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence4.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence5.png b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence5.png new file mode 100644 index 0000000000..c6ed7363be Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence5.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence6.png b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence6.png new file mode 100644 index 0000000000..1c172fbe60 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/updateDocSequence6.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/vbucket-partition-1.png b/content/understanding-couchbase/buckets-memory-and-storage/images/vbucket-partition-1.png new file mode 100644 index 0000000000..a5bc108355 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/vbucket-partition-1.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/vbucket-partitions-2.png b/content/understanding-couchbase/buckets-memory-and-storage/images/vbucket-partitions-2.png new file mode 100644 index 0000000000..a046629a7c Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/vbucket-partitions-2.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/images/vbucketToNodeMapping.png b/content/understanding-couchbase/buckets-memory-and-storage/images/vbucketToNodeMapping.png new file mode 100644 index 0000000000..acb8802855 Binary files /dev/null and b/content/understanding-couchbase/buckets-memory-and-storage/images/vbucketToNodeMapping.png differ diff --git a/content/understanding-couchbase/buckets-memory-and-storage/memory-and-storage.dita b/content/understanding-couchbase/buckets-memory-and-storage/memory-and-storage.dita new file mode 100644 index 0000000000..0417addf6c --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/memory-and-storage.dita @@ -0,0 +1,242 @@ + + + + + + Memory and Storage + + + + To facilitate high-speed data-access, Couchbase Server provides a caching layer and tunable disk I/O + priorities. + + + + +
    + + + Caching and Persistence + + +

    + Couchbase Server provides a fully integrated caching layer, which provides high-speed data-access. Couchbase + Server automatically manages the caching layer, ensuring that sufficient memory is available in relation + to occupied disk-space, in order to maintain optimal performance. Data associated with Couchbase + (as opposed to Ephemeral) buckets + is also maintained persistently, on disk. + Such items, when they come into the caching layer, are + placed on the disk queue, to be written to disk; and at the same time, if appropriate, are placed on a + replication queue, so that one or more replica buckets can be created or updated. +

    + +

    + Memory quotas, established by the Full Administrator when the server is configured, are automatically managed + by Couchbase Server with reference to watermarks, which specify how much free memory should be + consistently maintained within the caching layer. Consequently, infrequently used items are written to disk; and are + then removed from memory in order to free up space. This process, referred to + as ejection, is managed asynchronously, while the server continues to service active requests. + Items ejected from Couchbase buckets can subsequently be re-acquired from disk, as necessary. + A working-set of most frequently used data is tracked and maintained, and the items kept in memory to ensure high + performance. The priority of disk I/O is configurable per bucket. +

    + +
    + +
    + + + Saving New Items + + +

    + When a new item, such as a document, is saved by an application, it is + saved in memory only, if the item is contained by an Ephemeral bucket; + and is saved both in memory and on disk, if the item is contained by a + Couchbase bucket. If the bucket (either Ephemeral or Couchbase) has + been replicated one or more times, a copy of the new item is saved + in each replica. +

    + +

    + The following sequence illustrates how a new document is created by an application + and saved in a Couchbase bucket. It is assumed that the bucket has at least one + replica on the cluster. +

    + +

    +

    + +
      +
    1. + The application creates a new document. + +

      + +

      + +

      + +

      +
      2. + +
    2. + The application saves the document on Couchbase Server. The document is received in memory. + +

      + +

      + +

      + +

      +
      3. + +
    3. + Couchbase Server places a compressed copy of the document onto the Disk Queue so that + the document can be written to the disk; and a further copy onto the Replication Queue, + so that the document can be written to the replica bucket. (The copy placed on + the Replication Queue may or may not be compressed, depending on the compression + mode configured for the bucket.) + +

      + +

      + +

      + +

      +
      4. + +
    4. + Once written, the new document resides both in the memory and on the disk of the + node. It will also reside in the memory and on the disk of whichever other + nodes maintain the replicas of its buckets. + +

      + +

      + +

      + On each node containing the bucket or a replica, the document remains in memory; + unless it is ejected at some point, after which it remains on disk. +

      +

      + +

      +
      5. + +
    + +
    + +
    + + Updating Items + + +

    + Items that reside in the memory of Couchbase Server can be updated. If the item + belongs to a Couchbase bucket, the item is also updated on disk. If the item is + not currently in memory, but resides on disk, Couchbase Server retrieves the item, + so that it can be updated. This is demonstrated by the following sequence. +

    + +

    +

    + +
      +
    1. + The application provides an update to an existing document. + +

      + +

      + +

      + +

      + +
      2. + +
    2. + Since the document is not currently in memory, Couchbase Server seeks it + on disk, where is resides in compressed form. + +

      + +

      + +

      + +

      + +
      3. + +
    3. + The compressed document is retrieved, brought into memory, and decompressed. + +

      + +

      + +

      + +

      + +
      4. + +
    4. + The application's updates are applied to the uncompressed document. + +

      + +

      + +

      + +

      + +
      5. + +
    5. + The updated document is placed (in either compressed or uncompressed form, as appropriate) on + the replication queue, so that replicas + can be updated. The updated document is also re-compressed, and written to disk + locally. + +

      + +

      + +

      + +

      + +
      6. + +
    6. + The updated document is now retained locally on disk and in memory. The document + remains in memory unless it is ejected at some point, after which it + continues to reside on disk. + +

      + +

      + +

      + +

      + +
      7. + +
    + + +
    + + + + + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/memory.dita b/content/understanding-couchbase/buckets-memory-and-storage/memory.dita new file mode 100644 index 0000000000..32da1f0cc4 --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/memory.dita @@ -0,0 +1,275 @@ + + + + + + Memory + + + + Couchbase Server memory-management ensures high performance and scalability. + + + + +
    + + Service Memory Quotas + +

    + Memory-quota allocation on Couchbase Server occurs per service (except in the case + of the + Query Service, + which does not require a specific allocation). This allows + the availability of memory-resources to be tuned according to the assignment of services, + node by node. Note that the + Data Service + must run on at least one node; and that on each of those nodes, quotas for buckets, + specified at the time of bucket-creation, are subtracted + from the quota allocated to the Data Service. +

    + +

    + The memory-quota allocation specified for a given service applies to every instance of that + service across the cluster. For example, if 2048 MB is specified for the + Analytics Service, + and the Analytics Service is run on three of a cluster's five nodes, each of the three + instances of the service is duly allocated 2048 MB. Note that it is not possible to have + different memory allocations across multiple instances of the same service within a single cluster. +

    + +

    + By default, Couchbase Server allows 80% of a node's total available memory to be allocated to + the server and its services. + Consequently, if a node's total available memory is 100 GB, any attempt to allocate memory beyond 80 GB produces + an error. +

    + +

    + Instructions on how to allocate memory quotas to services when initializing a new cluster can be + found in the section + Configure Couchbase Server, + on the page + Initializing the Cluster. +

    + +

    + When a node is added to a cluster, the Default Configuration, as established by the set-up of + the first node in the cluster, is available: this covers all configurable elements, including memory + quotas. However, if insufficient memory for the default configuration is available on the new node, + the default configuration is prohibited: in such cases, settings + for the new node can be custom-configured, allowing an appropriate subset of services to be specified. +

    + +

    + If, when the initial node of a cluster is set up, only a subset of services is assigned, additional + nodes can subsequently be added, in order to host additional services: in which case, each new service can be given + its initial memory allocation as its node is added. +

    + +

    + Instructions on how to add a new node to a cluster, determine which services be can be retained, and + add new services, can + be found in the section + Join an Existing Cluster, + on the page + Initializing the Cluster. +

    + +

    + The Service Memory Quotas panel on the Settings screen of Couchbase Web Console lists all + services running on the cluster, and specifies + the memory allocation for each. + The panel is interactive, and allows the memory allocations to be changed and saved. If a modification + attemptedly exceeds a permitted maximum or minimum, + a notification + of the error is displayed, and the modification is disallowed. See + Configure Cluster Settings + for further information. +

    + +

    + Minimum-required memory-quotas are: +

    + + + + + + + + + Service + Minimum Memory Quota (in MB) + + + + + + Data + 256 + + + + Index + 256 + + + + Search + 256 + + + + Analytics + 1024 + + + + Eventing + 256 + + + + +
    + + +
    + + +
    + + + Initialization and Warmup + + +

    + When Couchbase Server is restarted on a node, the node goes through a warmup process + before it restarts the handling of data requests. During this warmup process, data + on disk is sequentially reloaded into memory. The time required for the reload + depends on the size and configuration of the system, the amount of data persisted on + the node, and the ejection policy configured for the buckets. +

    + +

    + Frequently used items are + identified via a scanner process, which examines an access log, and obtains the + appropriate keys. Corresponding items are then loaded with the highest priority. The scanner + process is configurable, via the CLI utility cbepctl, with the flush_param + parameter. + This utility also provides the parameters warmup_min_memory_threshold and + warmup_min_item_threshold, which can be used to schedule the resumption of traffic + before all items have been reloaded into memory. See + set flush_param. +

    + +
    + +
    + + + Ejection + + +

    + If a bucket's memory quota is exceeded, items may be ejected from the bucket by the + Data Service. Different + ejection methods are available, and are configured per bucket. Note that in + some cases, ejection is configured not to occur. + For detailed information, see + Buckets. +

    + +

    + For each bucket, available memory is managed according to two watermarks, which are + mem_low_wat and mem_high_wat. + If data is continuously loaded into the bucket, its quantity eventually + increases to the value indicated by the + mem_low_wat watermark. At this point, paging is triggered, so + that + memory-availability is continuously optimized. Then, as still more data is + loaded, the data's quantity + increases to the value indicated by the mem_high_wat watermark. + If, based on the bucket's configuration, items can be ejected from the bucket, + the Data Service + ejects items from the bucket until + the quantity of data has decreased to the mem_low_wat watermark. + If the rate at which data continues to be loaded is greater than that at which it is being ejected, or + if ejection cannot be performed, the system displays an insufficient memory + notification until + sufficient memory is available. +

    + +

    + Items are selected for ejection based on metadata that each contains, indicating + whether the item can be classified as Not Recently Used (NRU). If an item has not been recently used, + it is a candidate for ejection. +

    + +

    + The relationship of mem_low_wat and mem_high_wat to the + bucket's overall memory quota is illustrated as follows: +

    + +

    + +

    + +

    + The default setting for mem_low_wat is 75%. The default setting for + mem_high_wat is 85%. These settings give items from active + vBuckets a 40% chance of ejection; and give items from replica vBuckets a + 60% chance of ejection. The default settings can be changed by means of the + cbepctl utility. See + set flush_param. + +

    + +
    + + +
    + + + Expiry Pager + +

    + Scans for items that have expired, and erases them from memory and disk; after + which, a tombstone remains for a default period of 3 days. The expiry pager runs + every 60 minutes by default: for information on changing the interval, see cbepctl + set flush_param. + For more information on item-deletion and tombstones, see + Expiration. +

    +
    + +
    + + + Active Memory Defragmenter + + +

    + Over time, Couchbase Server-memory can become + fragmented. + Each page in memory is typically responsible for holding documents of a specific + size-range. Over time, if memory pages assigned to a specific size-range become sparsely populated + (due to documents of that size being ejected, or to items changing in size), the unused space in + those pages cannot be used for documents of other sizes, until a complete page is free, and that page + is re-assigned to a new size. Such effects, which are highly workload-dependent, may result in memory + that cannot be used efficiently. +

    + +

    + Couchbase Server provides an Active Memory Defragmenter, which periodically scans the + cache, to identify pages that are sparsely used. It then repacks the items on those pages, to + free up space. +

    + +
    + + + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/storage.dita b/content/understanding-couchbase/buckets-memory-and-storage/storage.dita new file mode 100644 index 0000000000..b063592d2e --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/storage.dita @@ -0,0 +1,164 @@ + + + + + + Storage + + + + Couchbase Server provides persistence, whereby certain items are stored on disk as well as + in memory; and reliability is thereby enhanced. + + + + +
    + + + Understanding Couchbase Storage + + +

    + Couchbase Server stores certain items in + compressed form on disk; and, whenever required, removes them. This allows data-sets to + exceed the size permitted + by existing memory-resources; since undeleted items not currently in memory can be + restored to memory from disk, as needed. It also facilitates backup-and-restore procedures. +

    + +

    + Generally, a client's interactions with the server are not blocked during + disk-access procedures. + However, if a specific item is being restored from disk to memory, the item + is not made available to the client until the item's restoration is complete. +

    + +

    + Not all items are written to disk: Ephemeral buckets and their items are maintained + in memory only. See + Buckets + for information. +

    + +

    + Items written to disk are always written in compressed form. Based on bucket configuration, + items may be maintained in compressed form in memory also. See + Compression + for information. +

    + +

    + Items can be removed from disk based on a configured point of expiration, referred to as + Time-To-Live. See + Expiration + for information. +

    + +

    + For illustrations of how Couchbase Server saves new and updates existing Couchbase-bucket items, + thereby employing both memory and storage resources, see + Memory + and Storage. +

    + +
    + +
    + + + Threading + + +

    + Synchronized, multi-threaded readers and writers provide simultaneous, high-performance + read-write operations for data on disk. Conflicts are avoided by assigning each thread (reader + or writer) a specific subset of the 1024 + vBuckets for each Couchbase bucket. For example, the first of five reader-threads is assigned + vBuckets 0, 5, 10, 15, 20, and so on; while the second is assigned vBuckets 1, 6, 11, 16, 21, and + so on. Thread-status can be viewed, by means of the cbstats command, specified with the + raw workload option. +

    + +

    + See + cbstats and + vBuckets for information. +

    + +
    + +
    + + + Deletion + + +

    + Items can be deleted by a client application: either by immediate action, or by setting + a Time-To-Live (TTL) value: this value is established through accessing the + TTL metadata field of the item, which establishes a future point-in-time + for the item's expiration. When the point-in-time is reached, Couchbase Server deletes + the item. +

    + +

    + Following deletion by either method, a tombstone is maintained by Couchbase Server, + as a record (see below). +

    + +

    + An item's TTL can be established either directly on the item itself, or via the bucket that + contains the item. For information, see + Expiration. +

    + +
    + +
    + + Tombstones + + +

    + A tombstone is the record of a deleted item. Each tombstone includes the item's key + and metadata. Tombstones are maintained in order to provide eventual consistency both between + nodes and between clusters. +

    + +

    + The Metadata Purge Interval establishes the frequency with which Couchbase Server purges + itself of tombstones: which means, removes them fully and finally. + The Metadata Purge Interval setting runs as part of auto-compaction. +

    + +

    + For more information, see + Post-Expiration Purging, + in + Expiration. +

    + +
    + +
    + + + Disk I/O Priority + + +

    + Disk I/O — reading items from and writing them to disk — does not block client-interactions: + disk I/O is thus considered a background task. The priority of disk I/O (along with that of + other background tasks, such as item-paging and DCP stream-processing) is configurable + per bucket. This means, for example, that one bucket's disk I/O can be granted + priority over another's. + For further information, see + Create a Bucket. +

    + +
    + + +     diff --git a/content/understanding-couchbase/buckets-memory-and-storage/vbuckets.dita b/content/understanding-couchbase/buckets-memory-and-storage/vbuckets.dita new file mode 100644 index 0000000000..c13f97593a --- /dev/null +++ b/content/understanding-couchbase/buckets-memory-and-storage/vbuckets.dita @@ -0,0 +1,92 @@ + + + + + vBuckets + + + vBuckets are virtual buckets that help distribute data effectively across a cluster, and + support replication across multiple nodes. + + + + +
    + + + Understanding vBuckets + + +

    + Couchbase Server allows users and applications to save data, in binary or JSON format, in named buckets. Each + bucket therefore contains keys and associated values. See + Buckets, for detailed + information. +

    + +

    + Within the memory and storage management system of Couchbase Server, both Couchbase and Ephermal buckets are implemented + as vBuckets, 1024 of which are created for every bucket (except on MacOS, where the number is 64). + vBuckets are distributed evenly across the memory and storage facilities of the cluster; and the bucket's items are + distributed evenly across its vBuckets. This evenness of distribution ensures that all instances + of the + Data Service + take an approximately equal share of the workload, in terms of numbers of documents to maintain, and + operations to handle. +

    + +

    + The 1024 vBuckets that implement a defined bucket are referred to as active vBuckets. + If a bucket is replicated, each replica is implemented as a further 1024 (or 64) vBuckets, referred to + as replica vBuckets. Thus, a bucket configured on Linux + to be replicated twice results in a total of 3072 vBuckets, distributed across the cluster. + Write operations are performed only on active vBuckets. Most read operations + are performed on active vBuckets, though items can also be read from replica vBuckets + when necessary. +

    + +

    + Note that vBuckets are sometimes referred to as shards. +

    + +

    + Items are written to and retrieved from vBuckets by means of a CRC32 hashing algorithm, which is + applied to the item's key, and so produces the number of the vBucket in which the item resides. vBuckets are + mapped to individual nodes by the Cluster Manager: the mapping is constantly updated and made generally available + to SDK and other clients. +

    + +

    + The relationships between a bucket, its keys (and their associated values), the hashing algorithm, vBuckets, server-mappings, + and servers, is illustrated below: +

    + +

    + +

    + +

    + Thus, an authorized client attempting to access data performs a hash operation on the appropriate key, and thereby + calculates the number of the vBucket that owns the key. The client then examines the vBucket map to determine the + server-node on which the vBucket resides; and finally performs its operation directly on that server-node. +

    + +

    + When a cluster-configuration changes — due to rebalance, failover, or node-addition — replica buckets + are promoted to primaries if appropriate, and both primaries and replicas are redistributed across the available + nodes of the modified cluster. The vBucket map is duly updated by the Cluster Manager. + The updated map is then sent to all cluster-participants. For additional information on the distribution + of vBuckets across the cluster, see + Availability. +

    + +

    + Note that this use of client-side hashing for access to Couchbase and Ephemeral bucket-data contrasts with the + Memcached data-access method; which requires active management of the server-list, and a specific + hashing algorithm, such as Ketama, to handle topology-changes. +

    + +
    + + +     diff --git a/content/understanding-couchbase/clusters-and-availability/cluster-manager.dita b/content/understanding-couchbase/clusters-and-availability/cluster-manager.dita new file mode 100644 index 0000000000..6815215253 --- /dev/null +++ b/content/understanding-couchbase/clusters-and-availability/cluster-manager.dita @@ -0,0 +1,386 @@ + + + + + + Cluster Manager + + + + The Couchbase Cluster Manager runs on all the nodes of a cluster, + maintaining essential per-node processes, and coordinating cluster-wide operations. + + + + +
    + + + Cluster Manager Architecture + + +

    + The architecture of the Cluster Manager is as follows: +

    + +

    + +

    + +

    + As shown, the Cluster Manager consists of two processes. The first, the + babysitter, is responsible for maintaining a variety of Couchbase Server-processes, + which indeed include the second Cluster Manager process, ns-server. The babysitter starts and monitors + all of these processes, logging their output to the file babysitter.log (see + Logging, for information + on logfile-locations). If any of the + processes dies, the babysitter restarts it. The babysitter is not cluster-aware. +

    + +

    + The processes for which the babysitter is responsible are: +

    + +
      +
    • + ns-server: Manages the node's participation in the cluster, as described in + ns-server, below. +

      + +

      +
      • + +
    • + kv engine: Runs as part of the + Data Service, which + must be installed on at least one cluster-node. Provides access to + Data. +

      + +

      +
      • + +
    • + services: One or more Couchbase + Services that + optionally run on the node. +

      + +

      +
      • + +
    • + xdcr: The program for handling Cross Data-Center Replication (XDCR). This is installed with + the Data Service, but + runs as an independent OS-level process, separate from the Cluster Manager itself. See + Availability, + for information. +

      + +

      +
      • + +
    • + view engine: The program for handling Views. This is installed with the Data Service, but + runs as an independent OS-level process, separate from the Cluster Manager itself. See + Views, for more + information. +

      + +

      +
      • + +
    • + other: Various ancillary programs. +

      + +

      +
      • + +
    + +
    + +
    + + + ns-server + + +

    + The principal Cluster-Manager process is ns-server, whose architecture is as follows: +

    + +

    + +

    + + +

    + The modules are: +

    + +

    + +

    + +
      + +
    • + REST Administration (UI and CLI): Supports administration of Couchbase Server, by means + of a REST API; which itself underlies both the user interface + provided by Couchbase Web Console, and the Couchbase Command-Line Interface. + +

      + +

      + +
      • + +
    • + Authentication: Protects node-resources with Role-Based Access Control. This is based + on credentials (usernames and passwords) associated with system-defined roles, each + of which is associated with a range of privileges. For detailed information, see + Authorization. + +

      + +

      + +
      • + +
    • + Master Services: Manages cluster-wide operations; such as master and replica + vBucket-placement, failover, node addition and subtraction, rebalance, cluster configuration-mapping, and + aggregation of statistical data. Note + that at any given time, only one of the instances of Master Services + on a multi-node cluster is in charge: the instances having negotiated among themselves, to identify and elect the + instance. Should the elected instance subsequently become unavailable, another takes over. The Master Services + are sometimes referred to as the Orchestrator. + +

      + +

      + +
      • + +
    • + Per Node Services: Manages the health of the current node, and handles + the monitoring and restart of its processes and services. + +

      + +

      + +
      • + +
    • + Per Node Bucket Services: Manages bucket-level operations for the current node; supporting replication, + fail-over, restart, and statistics-collection. + +

      + +

      + +
      • + +
    • + Generic Local Facilities: Provides local + configuration-management, libraries, workqueues, logging, clocks, ids, and events. + +

      + +

      + +
      • + +
    • + Generic Distributed Facilities: Supports node-discovery, configuration-messaging and alerts, + replication, and heartbeat-transmission. + +

      + +

      + +
      • +
    + +
    + +
    + + Adding and Removing Nodes + + +

    + The elected Master Services of the Cluster Manager are responsible for cluster membership. When topology changes, + a set of operations is executed, + to accomplish redistribution while continuing to handle existing workloads. This is as follows: +

    + +

    +

    + +
      + +
    1. + The Master Services update the new nodes with the existing cluster + configuration. + +

      + +

      + +
      2. + +
    2. + The Master Services initiate rebalance, and recalculate the vBucket map. + +

      + +

      + +
      3. + +
    3. + The nodes that are to receive data initiate DCP replication-streams from the existing nodes for + each vBucket, and begin building new copies of those vBuckets. This occurs for both active and + replica vBuckets, depending on the new vBucket map layout. + +

      + +

      + +
      4. + +
    4. + Incrementally — as each new vBucket is populated, the data is replicated, and indexes are + updated — an atomic switchover takes place, from the old vBucket to the new vBucket. + +

      + +

      + +
      5. + +
    5. + As new vBuckets on new nodes become active, the Master Services ensure that the new vBucket + map and cluster topology are communicated to all nodes and clients. This process is repeated + until rebalance is complete. + +

      + +

      + +
      6. +
    + +

    + The process of removing one or more Data-Service nodes is similar to that of adding: + vBuckets are created on nodes that are to be maintained, and data is copied to them from vBuckets resident + on nodes that are to be + removed. When no more vBuckets remain on a node, the node is removed from the cluster. +

    + +

    + When adding or removing nodes that do not host the Data Service, no data is moved: therefore, + nodes are added or removed from the cluster map without data-transition. +

    + +

    + Once the process of adding or removing is complete, + and a new cluster map has been made available by the Master Services, client SDKs automatically begin load-balancing + across those services, using the new cluster map. +

    + +

    + For the practical steps to be following in adding and removing nodes, see + Adding a Node and + Removing a Node. + +

    + +
    + +
    + + Node-Failure Detection + +

    + Nodes within a + Couchbase Server-cluster provide status on their health by means of a heartbeat mechanism. + Heartbeats are provided by all instances of the Cluster Manager, at regular + intervals. Each heartbeat contains basic statistics on the node, which are used to assess the + node's condition. +

    + +

    + The Master Services keep track of heartbeats received from all other nodes. If automatic + failover is enabled, and no heartbeats are received from a node for longer than the default + timeout period, the Master Services may automatically fail the node over. +

    + +

    + For detailed information on failover options, see + Failing over a Node. +

    + +
    + + +
    + + + vBucket Distribution + + +

    + Couchbase Server buckets physically contain 1024 master and 0 or more replica + vBuckets. The Master Services govern the placement of + these vBuckets, to maximize availability to and rebalance performance. The vBucket map + is recalculated + whenever the cluster topology changes, by means of the following rules: +

    + +
      + +
    • + Master and replica vBuckets are placed on separate nodes. +
      • + +
    • + If a bucket is configured with more than one replica, each additional replica vBucket is + placed on a separate. +
      • + +
    • If Server Groups are defined for master vBuckets, the replica vBuckets are placed in + a separate groups. See , for more information.
      • + +
    + +
    + +
    + + + Centralized Management, Statistics, and Logging + + +

    + The Cluster Manager simplifies centralized management with centralized configuration-management, + statistics-gathering, and logging services. All configuration-changes are managed by the + Master Services, and are pushed out from the Master Services node to the other nodes. +

    + +

    + Statistics are accessible through all the Couchbase administration interfaces: The + Command Line Interface, + (specifically, the + cbstats tool), the + REST API, and + Couchbase Web Console. +

    + +
    + +     diff --git a/content/understanding-couchbase/clusters-and-availability/clusters-and-availability.dita b/content/understanding-couchbase/clusters-and-availability/clusters-and-availability.dita new file mode 100644 index 0000000000..81a41a8f24 --- /dev/null +++ b/content/understanding-couchbase/clusters-and-availability/clusters-and-availability.dita @@ -0,0 +1,82 @@ + + + + + + Clusters and Availability + + + + One or more instances of Couchbase Server constitute a cluster, which replicates data across + server-instances, and across clusters; and so ensures high availability. + + + + +
    + + + Clusters + + +

    + A Couchbase cluster consists of one or more instances of Couchbase Server, each + running on an independent node. Data and services are shared across the cluster. +

    + +

    + When Couchbase Server is being configured on a node, it can be specified either as its + own, new cluster, or as a participant in an existing cluster. Thus, once a cluster + exists, successive nodes can be added to it; each node running Couchbase Server. When + a cluster has multiple nodes, the Couchbase Cluster Manager runs on each node: + this manages communications between nodes, and ensures that all nodes are healthy. The + Cluster Manager provides information on the cluster to the user interface of + Couchbase Web Console. +

    + +

    + Services can be configured to run on all or some nodes in a cluster. For example, given + a cluster of five nodes, a small dataset might require the Data Service on only one of + the nodes; a large on four or five. Alternatively, a heavy query workload might require the + Query Service to run on multiple nodes, rather than just one. This ability to scale + services individually promotes optimal hardware-resource utilization. +

    + +
    + +
    + + Availability + + +

    + Data is automatically distributed across a cluster by Couchbase Server: applications + are not involved. Each defined bucket is stored by the Data Service as 1024 + vBuckets (virtual buckets), which are spread evenly across all + available Data Service nodes. Documents are stored intact within vBuckets. + vBuckets can also be replicated, across the cluster, by means of the + Database Change Protocol; with each replica always existing + on a node different from that of its original. +

    + +

    + Couchbase Server automatically handles the addition and removal of nodes, and the + failure of + nodes; such that no data-loss occurs. vBuckets and their replicas are + redistributed across available nodes whenever a change of configuration + is detected. +

    + +

    + High Availability is achieved by means of Cross Datacenter Replication + (XDCR); whereby the contents of a bucket can be selectively replicated to a bucket + maintained on a remote cluster. Additionally, Server Group + Awareness allows groups of nodes to be defined, within the cluster, to + protect the cluster against significant infrastructure-failure.

    + + +
    + + + +     diff --git a/content/understanding-couchbase/clusters-and-availability/connectivity-architecture.dita b/content/understanding-couchbase/clusters-and-availability/connectivity-architecture.dita new file mode 100644 index 0000000000..a7772a6dc2 --- /dev/null +++ b/content/understanding-couchbase/clusters-and-availability/connectivity-architecture.dita @@ -0,0 +1,207 @@ + + + + + + Connectivity + + + + Couchbase Server handles client to cluster, node to node, and cluster to cluster communications. It + also provides connectivity to a number of third-party products. + + + +
    + + + Client to Cluster + + +

    + Client applications communicate with a Couchbase Server-cluster through a set of server-defined access-points, + each of which provides ports for both clear and encrypted communication. +

    + +

    + The principal access-points for clients are as follows: +

    + + + + + + + + + + + Type + Port + API + Description + + + + + REST + 8091, 18091 (SSL) + REST API administrative operations + Direct connection to a single node in the cluster, to perform administrative operations, + monitoring, and alerting; by means of the + REST API. + + + + ONLINE + 11210, 11207 (SSL) + Core Data operations + Stateful connections from a client to cluster-nodes running the + Data Service for + CRUD operations. + + + REST + 8093, 18093 (SSL) + Query with N1QL API + Load-balanced connection across cluster-nodes running the + Query Service for + N1QL queries. + + + + REST + 8094, 18094 (SSL) + Search Service + Load-balanced connections across cluster-nodes running the + Search Service for + Full Text Search queries. + + + REST + 8095, 18095 (SSL) + Analytics Service + Load-balanced connections across cluster-nodes running the + Analytics Service. + + + REST + 8092, 18092 (SSL) + Query with View and Spatial View API + Load-balanced connection across cluster-nodes running the + Data Service for + View queries. + + + +
    + +

    + Note that the above table lists only a subset of Couchbase Server network ports. For a complete list, see + . +

    + +
    + +
    + + Node to Node + + + Cluster-nodes intercommunicate in order to + replicate data, maintain indexes, check the health of nodes, communicate changes to the + cluster-configuration, and more. See + Cluster Manager, + for detailed information. + +
    + +
    + + + Cluster to Cluster + + +

    + Couchbase Server-clusters communicate + with one another by means of Cross Datacenter Replication (XDCR). See + Availability, + for detailed information. +

    + +
    + +
    + + + Cluster to Connector + + +

    + Couchbase Server-clusters communicate with third party products; by means of connectors. + Connectors are provided for Elasticsearch, Hadoop, Kafka, Spark, and Talend. Drivers + are provided for ODBC and JDBC. See + , for + detailed information. +

    + +
    + +
    + + + Connectivity Phases + + +

    + Client connectivity is established in three phases: Authentication and Authorization, Discovery, and + Service Connection. +

    + +

    + +

    + +
      + +
    1. + Authentication and Authorization: The client authenticates with username and password. If + these are associated with a Couchbase-Server defined role, itself associated with appropriate + privileges on the resource access to which is being requested, the client is authorized, and access is granted. Otherwise, + access is denied. See + Authorization + for details. + +

      + +

      +
      2. + +
    2. + Discovery: A cluster-map is returned to the client. This indicates the current + cluster-topology; including the list of nodes, the data-distribution across the nodes, and + the service-distribution accross the nodes. + +

      + +

      +
      3. + +
    3. + Service Connection: Once in possession of the cluster-map, the client determines the + connections needed to establish and perform service-level operations. Additional + authorizations may be required, depending on the operations being attempted. Note that in the event of + topology-changes, a service connection-request may result in an exception; in which case discovery + must be re-run, and operations retried with new connections. + +

      + +

      +
      4. + +
    + + +
    + +     diff --git a/content/understanding-couchbase/clusters-and-availability/groups.dita b/content/understanding-couchbase/clusters-and-availability/groups.dita new file mode 100644 index 0000000000..8cadff84a0 --- /dev/null +++ b/content/understanding-couchbase/clusters-and-availability/groups.dita @@ -0,0 +1,132 @@ + + + + + Server Group Awareness + + + Individual server-nodes can be assigned to specific groups, within + a Couchbase Cluster. This allows active vBuckets to be maintained on a different group + from that of their corresponding replica vBuckets; so that + if a group goes offline, bucket-data remains available on another group. + + + + Server groups and the Server Group Awareness feature are only available in + Couchbase Server Enterprise Edition. +

    + Server Group Awareness provides enhanced availability. Specifically, it protects a cluster + from large-scale infrastructure failure, through the definition of groups. Each group is + created by an appropriately authorized administrator, and specified to contain a subset of the + nodes within a Couchbase Cluster. Following group-definition and rebalance, the active vBuckets + for any defined bucket are located on one group, while the corresponding replicas are located on + another group. This allows Group Failover to be enabled, so that if an entire group goes + offline, its replica vBuckets, which remain available on another group, can be automatically + promoted to active status.

    +

    Groups should be defined in accordance with the physical distribution of cluster-nodes. For + example, a group should only include the nodes that are in a single server rack, or in the + case of cloud deployments, a single availability zone. Thus, if the server rack or + availability zone becomes unavailable due to a power or network failure, Group Failover, if + enabled, allows continued access to the affected data.

    +

    Data-protection is optimal when groups are assigned equal numbers of nodes, and + vBuckets are therefore distributed such that none ever occupies the same group as its associated + active vBucket. By contrast, when groups are not assigned equal numbers of nodes, + rebalance can only produce a best effort redistribution of replica vBuckets: this may + result in replica vBuckets occupying the same group as their associated active vBuckets; meaning + that data may be lost if such a group becomes unavailable.

    +

    General notes about Server Group Awareness: +

    + +
      +
    • Server Group Awareness only applies to the Data Service.
      • +
    • If a cluster contains only one node, or only one group of nodes, enabling Group Failover has + no effect; meaning that failover can only be performed on a per-node basis.
      • +
    • Failover should be enabled for server groups only if three or more server groups have been + established, and sufficient capacity exists to absorb the load of any failed-over group.
      • +
    • When you initialize a new Couchbase Server cluster, the first node, and all subsequent + nodes, are automatically placed in a server group named Group 1. Once you create additional + server groups, you are required to specify a server group when adding additional cluster + nodes.
      • +
    + +

    For information on failing over individual nodes, see Failing over a Node.

    +

    For information on vBuckets, see Buckets.

    +

    For information on the standard (non-Group-based) distribution of replica vBuckets across a + cluster, see Availability.

    +
    + Equal Groups +

    The following illustration shows how vBuckets are distributed across two groups; each group + containing four of its cluster's eight nodes.

    +

    + +

    +

    Note that Group 2 contains all the replica vBuckets that correspond to active vBuckets on + Group 1; while conversely, Group 1 contains all the replica vBuckets that correspond to active + vBuckets on Group 2.

    +
    +
    + Unequal Groups +

    The following illustration shows how vBuckets are distributed across two groups: Group 1 + contains four nodes, while Group 2 contains five.

    +

    + +

    +

    Group 1 contains all the replica vBuckets that correspond to active vBuckets on Group 2. + However, since the groups contain unequal number of nodes, Group 2 not only contains all the + replica vBuckets that correspond to active vBuckets on Group 1, but also contains all the + replica vBuckets for its own additional node, Server 9 — the replicas for Server 9 being + distributed across the other Group 2 nodes; which are Servers 5, 6, 7, and 8. Server 9 contains + its own active vBuckets, plus replica vBuckets for Group 1.

    +

    This means that if Group 2 were to go offline, Group Failover would not preserve the + replica vBuckets for Server 9, since these only existed on Group 2 itself.

    +
    +
    + Node-Failover Across Groups +

    When an individual node within a group goes offline, rebalance provides a best effort + redistribution of replica vBuckets. This keeps all data available, but results in some data + being no longer protected by the Groups mechanism. This is shown by the following illustration, + in which Server 2, in Group 1, has gone offline, and a rebalance and failover have occurred.

    +

    + +

    +

    With the active vBuckets on Server 2 no longer accessible, the replica vBuckets for Server 2 + have been promoted to active status, on the servers of Group 2. The data originally active on + Server 2 is thereby kept available. Note, however, that if Group 2 were now to go offline, the + data originally active on Server 2 would be lost, since it now exists only on Group 2 servers. +

    +
    +
    + Defining Groups and Enabling Group Failover +

    To define and manage groups:

    +
      +
    • With Couchbase Web Console, see Manage Server Groups.

      +
      • +
    • With CLI, see group-manage.

      +
      • +
    • With the REST API, see Server Groups API.
      • +
    +

    To enable Group Failover:

    +
      +
    • With Couchbase Web Console, see Node Availability.

      +
      • +
    • With CLI, see setting-autofailover.

      +
      • +
    • With the REST API, see Enabling and Disabling Auto-Failover.

      +
      • +
    +
    + + +     diff --git a/content/understanding-couchbase/clusters-and-availability/images/cluster-manager-internals.png b/content/understanding-couchbase/clusters-and-availability/images/cluster-manager-internals.png new file mode 100644 index 0000000000..04ad4862f8 Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/cluster-manager-internals.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/clusterManagerArchitecture.png b/content/understanding-couchbase/clusters-and-availability/images/clusterManagerArchitecture.png new file mode 100644 index 0000000000..6defd8bfed Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/clusterManagerArchitecture.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/clusterManagerArchitecture2.png b/content/understanding-couchbase/clusters-and-availability/images/clusterManagerArchitecture2.png new file mode 100644 index 0000000000..420fc3c522 Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/clusterManagerArchitecture2.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/groups-two-equal.png b/content/understanding-couchbase/clusters-and-availability/images/groups-two-equal.png new file mode 100644 index 0000000000..c62ad16037 Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/groups-two-equal.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/groups-two-failover-one-node.png b/content/understanding-couchbase/clusters-and-availability/images/groups-two-failover-one-node.png new file mode 100644 index 0000000000..3ef6d2c8bb Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/groups-two-failover-one-node.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/groups-two-unequal.png b/content/understanding-couchbase/clusters-and-availability/images/groups-two-unequal.png new file mode 100644 index 0000000000..5fe4450664 Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/groups-two-unequal.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/nsServerArchitecture.png b/content/understanding-couchbase/clusters-and-availability/images/nsServerArchitecture.png new file mode 100644 index 0000000000..5f1cccce82 Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/nsServerArchitecture.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/nsServerArchitecture2.png b/content/understanding-couchbase/clusters-and-availability/images/nsServerArchitecture2.png new file mode 100644 index 0000000000..7b14afab0b Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/nsServerArchitecture2.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/server-replication.png b/content/understanding-couchbase/clusters-and-availability/images/server-replication.png new file mode 100644 index 0000000000..ac5e5c7f7a Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/server-replication.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/vBucketReplication.png b/content/understanding-couchbase/clusters-and-availability/images/vBucketReplication.png new file mode 100644 index 0000000000..9cc1a10c82 Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/vBucketReplication.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/images/xdcr.png b/content/understanding-couchbase/clusters-and-availability/images/xdcr.png new file mode 100644 index 0000000000..dc98e71658 Binary files /dev/null and b/content/understanding-couchbase/clusters-and-availability/images/xdcr.png differ diff --git a/content/understanding-couchbase/clusters-and-availability/replication-architecture.dita b/content/understanding-couchbase/clusters-and-availability/replication-architecture.dita new file mode 100644 index 0000000000..818fef68b0 --- /dev/null +++ b/content/understanding-couchbase/clusters-and-availability/replication-architecture.dita @@ -0,0 +1,337 @@ + + + + + + Availability + + + + Couchbase Server ensures the availability of data across the nodes of a cluster; across + groups of nodes within the cluster; and + across separate clusters, potentially located in different data-centers. + + + +
    + + + Intra-cluster Replication + + +

    + Intra-cluster replication involves replicating data across the node of a cluster. +

    + +

    + Couchbase + Data is defined logically to + reside in + Buckets. + Each bucket, when defined, is implemented by Couchbase Server as + vBuckets, + up to 1024 of which thereby exist in memory (and, in the case of Couchbase Buckets, on disk); the + exact number at any given time depending on the number of items to be stored. Items are associated with + vBuckets by means of a hashing algorithm, and buckets are assigned to nodes according to a fixed + mapping, determined and updated by the Master Services of the + ns-server + component of the Cluster Manager. +

    + +

    + Up to three replica buckets can be defined for every bucket. Each replica itself is also implemented + as 1024 vBuckets. A vBucket that is part of the original implementation of the defined bucket is referred + to as an active vBucket. Therefore, a bucket defined with two replicas has 1024 active vBuckets + and 2048 replica vBuckets. Typically, only active vBuckets are accessed for read and write operations: + although vBuckets are able to support read requests. Nevertheless, + vBuckets receive a continuous stream of mutations from the active vBucket by means of the Database + Change Protocol (DCP), and are thereby kept constantly up to date. + To ensure maximum availability of data in case of node-failures, the Master Services for the cluster + calculate and implement the optimal + vBucket Distribution + across available nodes: consequently, the chance + of data-loss through the failure of an individual node is minimized, since + replicas are available on the nodes that remain. This is shown by the following illustration: +

    + + + +

    + The illustration shows active and replica vBuckets that correspond to a single, user-defined bucket, for + which a single replication-instance has been specified. The first nine active vBuckets are shown, along + with their nine corresponding, replica vBuckets; distributed across three server-nodes. The distribution of + vBuckets indicates a likely distribution calculated by Couchbase Server: no replica resides on the same + node as its active equivalent: therefore, should any one of the three nodes fail, its data remains + available. +

    + +

    + When a node becomes unavailable, + failover + can be performed: meaning that the Cluster Manager is instructed to + read and write data only on those cluster-nodes that are still + available. Failover can be performed by manual intervention, or automatically: + as part of this process, where required, + replica vBuckets are promoted to active status. Automatic failover can be performed only + for one node at a time, and only up to a configurable number of times, the maximum being three. +

    + +

    + Note that the Cluster Manager never performs automatic failover where data-loss might result. The + number of times failover can be safely performed depends on how many nodes and replicas exist. For + example, in a five node cluster with one replica, a single node can be failed over without danger: + if a second node fails, failover might result in data-loss, due to required replicas no longer + being available. Similarly, in a five node cluster with two replicas, two nodes can be failed over + without danger: a third cannot. In such cases, a slower, manual recovery-process is required. See + Failing over a Node for more + information. +

    + +

    + For information on configuring replicas, see + Create a Bucket. + As a rule, configure one replica for a cluster of up to five nodes; one or two replicas for + from five to ten nodes; and one, two, or three replicas for over ten nodes. +

    + +

    + Note that intra-cluster availability can further be optimized by defining + Groups. +

    +
    + +
    + + + Cross Data-Center Replication + + +

    + Cross Data-Center Replication (XDCR) replicates data between + clusters: this provides protection against data-center failure, and also provides + high-performance data-access for globally distributed, mission-critical applications. +

    + +

    + XDCR replicates data from a specific bucket on the source-cluster to a specific bucket on + the target-cluster. Data from the source-bucket is pushed + to the target-bucket by means of an XDCR agent, running on the source cluster, using the + Database Change Protocol. + Any bucket (Couchbase or Ephemeral) on any cluster can be specified as a source or a target + for one or more XDCR-definitions. +

    + +

    + The XDCR agent on the source node propagates + mutations from the source to the target vBucket. Since there + are equal numbers of vBuckets (default is 1024) on both the source and the destination clusters, + there is a one-to-one match for each source and target vBucket. Source and target clusters do not require + identical topology; since XDCR agents are topology aware. Complex cross-cluster topologies are + supported: including bidirectional, + ring tree, and structured. +

    + +

    + For more information, see + Managing XDCR. +

    + +
    + +
    + + + XDCR Conflict Resolution + + +

    + XDCR replication is asynchronous and does not preclude updates + from occurring independently on source and target clusters: this means that data-updates + must be coordinated, to prevent conflicts. +

    + +

    + Conflict is determined to occur when two or more copies of the same document differ, due + to being at different stages of update at a given time. + Two methods of conflict resolution are supported: +

    + +

    +

    + +
      +
    • + Sequence number: Compares the values in the documents' Revision ID meta-data fields to resolve + conflicts. This field tracks the number of mutations made to a document: the document-copy + with the most mutations is chosen to be the correct one, and others are updated to be identical + to it. +

      + +

      +
      • + +
    • + Timestamp: Compares the documents' timestamps to resolve + conflicts. This field shows how recently update occurred. The document-copy with + the highest (most recent) timestamp is chosen to be the correct one, and other are updated + to be identical to it. +

      + +

      +
      • + +
    + +

    + See for more details. +

    + +
    + +
    + + Database Change Protocol (DCP) + + +

    + Database Change Protocol (DCP) is the protocol used to stream bucket-level mutations. + DCP is used for high-speed replication of mutations; to maintain replica vBuckets, + incremental MapReduce and spatial views, Global Secondary Indexes (GSIs), XDCR, backups, + and external connections. +

    + +

    + DCP is a memory-based replication protocol that is ordering, resumable, and + consistent. DCP + streams changes made in memory to items, by means of a replication queue. +

    + +

    + DCP involves the following concepts: +

    + +

    +

    + +
      + +
    • + Application client: A client external to the cluster. Transmits read, write, + update, delete, and query requests to the server-cluster. + See + Compression, + for an illustration. + +

      + +

      +
      • + + +
    • + DCP client: A Couchbase client that is either internal or external + to the cluster. Streams data from one or more Couchbase Server-nodes, + in support of intra-cluster replication, indexing, XDCR, services, + incremental backup, connectors, and mobile synchronicity. See + Compression, + for an illustration. + +

      + +

      +
      • + +
    • + Failover log: A list of previously known vBucket-versions, for a vBucket. If a client connects + to a server, and was previously connected to a vBucket-version other than the current one, + the failure log is used to find a rollback point. + +

      + +

      +
      • + +
    • + History branch: If a replica vBucket does not contain the latest changes, but becomes active, + its record of changes (referred to as a history branch) is recognized by DCP clients; + and the vBucket duly re-versioned and + updated. Such action, if required, must occur before any further major operation (such as a node addition + or removal, or a swap-rebalance) can be performed. + +

      + +

      +
      • + +
    • + Mutation: An event that deletes a key, or changes the value to which a key points. Mutations + occur when transactions such as create, update, delete, or expire are executed. + + + +

      + +

      +
      • + +
    • + Rollback point: The commencement of a recognized history branch, indicating + the point in the mutation-sequence from which a vBucket must be brought up to date. + + +

      + +

      +
      • + +
    • + Sequence number: Given to each mutation to a vBucket, so that the sequence of mutations + is recorded, and can be referenced by interested processes. The later the mutation, the higher + the number. The sequence is strictly relevant to the vBucket, and does not provide a cluster-wide + ordering of events. + +

      + +

      +
      • + +
    • + Snapshot: A representation of the exact state of mutations on either a write queue or + in storage, provided by Couchbase Server to an interested client. + +

      + +

      +
      • + +
    • + vBucket stream: A grouping of messages related to receiving mutations for a specific vBucket. + This includes mutation, deletion, expiration, and snapshot-marker messages. + + +

      + +

      +
      • + +
    • + vBucket version: A Universally unique identifier (UUID) and sequence-number pair associated + with a vBucket. A new version is assigned to a vBucket by the new master-node whenever + a history branch is recognized. The UUID is a randomly generated number; and the sequence number is the + one last processed by the vBucket, at the time the version was created. + +

      + +

      +
      • + + +
    + + +
    + + + + +     diff --git a/content/understanding-couchbase/data/data.dita b/content/understanding-couchbase/data/data.dita new file mode 100644 index 0000000000..014ec83b49 --- /dev/null +++ b/content/understanding-couchbase/data/data.dita @@ -0,0 +1,367 @@ + + + + Data + + Couchbase Server saves data as items, each of which has a key and a value. + + + + +
    + + + Items, Keys, and Values + + +

    + Each item consists of a key and a value. Each key is unique + within its bucket, and values + can be either binary or JSON. +

    + +
    + +
    + + Keys + + +

    + Each value (binary or JSON) is identified by a unique key, defined by the user or + application when the item is saved. + The key is immutable: once the item is saved, the key cannot be changed. +

    + +

    + Note that Couchbase also refers to an item's key as its id. +

    + +

    + Each key: +

    + +

    +

    + +
      +
    • + Must be a UTF-8 string with no spaces. Special characters, such as (, %, + /, ", and _, are acceptable. +

      + +

      +
      • + +
    • + May be no longer than 250 bytes. +

      + +

      +
      • + +
    • + Must be unique within its bucket. +

      + +

      +
      • + +
    + +
    + +
    + + Values + + +

    + The maximum size of a value is 20 MiB. A value can be either: +

    + +
      +
    • + Binary: Any form of binary is acceptable. Note that a binary value cannnot be parsed, indexed, + or queried: it can only be retrieved by key. + +

      +

      + +
      • + +
    • + JSON: A JSON value, referred to as a document, can be parsed, indexed, and queried. + Each document consists of one or more attributes, + each of which has its own value. An attribute's value can be a basic type, such + as a number, string, or Boolean; or a complex, such as an embedded document or an array. + +

      + +

      +
      • + +
    + +
    + +
    + + Document Structure + + +

    + A sample JSON document is provided immediately below. Its contents are as follows: +

    + +

    +

    + +
      +
    • + a1 and a2 are attributes that respectively + have a single number and a single string as their values. + +

      + +

      +
      • + +
    • + a3 is an attribute whose value is an embedded document consisting of a single + attribute, b1, whose own value is an array of three numbers. + +

      + +

      + +
      • + +
    • + a4 is an attribute whose value is an array of two + documents, each document consisting an attribute (c1 or c2) + whose own values are respectively a string and a number. + +

      + +

      +
      • +
    + + { + "a1" : number, + "a2" : "string", + "a3" : { + "b1" : [ number, number, number ] + }, + "a4" : [ + { "c1" : "string", "c2" : number }, + { "c1" : "string", "c2" : number } + ] +} + +

    + Note that the official JSON definition is provided at + RFC 7159, and at + ECMA 404. +

    + +

    + The benefits offered to application-programmers by JSON are considered in + The Couchbase Data Model. +

    + +
    + +
    + + Sub-Documents + + +

    + A sub-document is an inner component of a JSON document. The sub-document is referred to by a path, + which specifies attributes and array-positions. +

    + +

    + The example given above includes the following paths: +

    + +

    +

    + +
      +
    • + a1, whose value is a number. +

      + +

      +
      • + +
    • + a3.b1, whose value is an array of numbers. +

      + +

      +
      • + +
    • + a3.b1[0], whose value is the first number in an array. + +

      +

      +
      • + +
    • + a4[1].c2, whose value is a number. +
      • +
    + +

    + The Couchbase SDK provides a Sub-Document API, which allows paths to be specified, + and sub-documents thereby read and written to. This makes it unnecessary to transfer entire + documents over the network when only partial modifications are required, potentially + providing significant improvements in network-performance. + For further information, see + Sub-Document Operations. +

    + +
    + +
    + + + Metadata + + +

    + Metadata is automatically generated and stored for each item saved in Couchbase Server. For + example, the following document, which contains airport-information, has been + saved with the key airport_1306: +

    + + { + "airportname": "Brienne Le Chateau", + "city": "Brienne-le Chateau", + "country": "France", + "faa": null, + "geo": { + "alt": 381, + "lat": 48.429764, + "lon": 4.482222 + }, + "icao": "LFFN", + "id": 1306, + "type": "airport", + "tz": "Europe/Paris" +} + +

    + Couchbase Server generates metadata in association with the document, when the document is saved. The metadata might + be as follows: +

    + + { + "meta": { + "id": "airport_1306", + "rev": "1-1526338d1bbb00000000000002000000", + "expiration": 0, + "flags": 33554432, + "type": "json" + }, + "xattrs": {} +} + +

    + The attributes within the metadata are: +

    + +

    + +

    + +
      + +
    • + meta: The attribute whose value is the standard metadata for the saved + document, airport-1306. +

      +
      • + +
    • + id: The key of the saved document, which is airport_1306. +

      +
      • + +
    • + rev: The revision or sequence number. This value is for internal server-use only: it is + used in the resolution of conflicts that occur when replicated documents are updated concurrently on different servers, + representing the number of times the document has been mutated. For more information, see + XDCR Conflict Resolution. +

      +
      • + +
    • + expiration: The expiration-time (or Time-To-Live) of the document. If non-zero, this + determines the point at which the document is removed from the system. The value can be set either + per-document, by means of Couchbase SDK APIs (which is referred to as TTL); or + per bucket, by means of Couchbase Web Console, the Couchbase CLI, or the + Couchbase REST API (which is referred to as Bucket TTL). + +

      + For information on TTL, see + Core Operations. For information + on Bucket TTL, see + Bucket TTL. +

      + +

      + +
      • + +
    • + flags: Couchbase SDK-specific values that may be used to identify the type of data saved, or to + specify formatting. + +

      +
      • + +
    • + type: The type of the saved value, which in this case is json. +

      +
      • + +
    • + xattrs: Extended Attributes, which constitute a special kind of metadata, some of + which is system-internal, some of which can optionally be written and read by user-applications. See + Extended Attributes for + more information. +

      +
      • + + +
    + +
    + +
    + + + Size Limits + + +

    + The following diagram indicates the respective maximum sizes of the components of a Couchbase Server data-item. +

    + +

    + +

    + + + +
    + + + +     diff --git a/content/understanding-couchbase/data/document-data-model.dita b/content/understanding-couchbase/data/document-data-model.dita new file mode 100644 index 0000000000..1aaff011f1 --- /dev/null +++ b/content/understanding-couchbase/data/document-data-model.dita @@ -0,0 +1,160 @@ + + + + The Couchbase Data Model + + + The Couchbase Data Model provides a lightweight, flexible schema; which can be progressively evolved + by applications, over time. + + + + +
    + + + Couchbase Server and JSON: The Benefits + + +

    + The Couchbase data model is based on JSON, which + provides a simple, lightweight, human-readable notation. + It supports basic data types, such as numbers and strings; and + complex types, such as embedded documents and arrays. JSON provides rapid serialization + and deserialization; is native to JavaScript; and constitutes the most common REST API return-type. + Consequently, JSON is extremely convenient for web-application programming. +

    + +

    + An individual document often represents a single instance of an object in application + code. A document might be considered equivalent to a row in a relational table; with each + of the document's attributes being equivalent to a column. Couchbase, however, provides greater + flexibility than relational databases, in that it can store JSON documents with varied schemas. +

    + +

    + Documents can contain nested structures. This allows + developers to express many-to-many relationships without requiring a reference + or junction table; and is naturally expressive of hierarchical data. +

    + +
    + +
    + + + Documents versus Tables + + +

    + The nature and value of the document data model is clarified by comparison with the + relational. To support an online flight-booking application, allowing users to search + for flights by date, the relational model requires multiple tables — for flights, airlines, + and schedules. The result may be as follows: +

    + +

    + +

    + +

    + By contrast, the document model likely requires only a single document, which embeds + an array of schedules for all flights between each of two airports: +

    + +

    + +

    + +

    + Thus, in the document model, each document can be highly self-contained. This supports the rapid + fulfillment of application-requests, and has important implications for both scalability and latency: + one document can be replicated, or atomically changed, without + other documents needing to be accessed; eradicating the need + for complex inter-node coordination, and minimizing contention. +

    + +
    + +
    + + + Flexible, Dynamic Schema + + +

    + In the document model, a schema is the result of an application's structuring of its + documents: schemas are entirely defined and managed by applications. A document's + structure consists of its inner arrangement of attribute-value pairs. +

    + +

    + Couchbase Server does not enforce uniformity: document-structures can vary, even across multiple + documents that each contain a type attribute with a common value. This allows + differences between objects to be represented + with great efficiency. It also allows a schema to be progressively evolved by an application, as required: + properties and structures can be added to the document, without other documents needing + to be updated in the same way. (This flexibility is especially advantageous when the application itself + is large and long-lived.) +

    + +
    + +
    + + + Document Design Considerations + + +

    + When JSON documents are initially designed, performance and scalability should be considered. The + best document-design can often be + determined through consideration of access-patterns and object-management + routines: +

    + +

    + +

    + +
      +
    • + The definition of a small number of rich documents, each embedding complex information, may sometimes be appropriate: in + order to group together properties typically accessed or written at the same time. This potentially allows + information + to be read or written in a single operation; atomicity to be improved, due to the simultaneous + occurrence of mutations; and scalability to be enhanced, due to the existence of fewer relations between + independent objects. It may also allow the grouped properties to be more easily maintained in a state of mutual + consistency. +

      + +

      +
      • + +
    • + The definition of a large number of simple documents, each of which refers to others, + may be appropriate when access-patterns are predictable, and data-size needs to be kept small, in order to reduce + network-bandwidth consumption. Documents can refer to each other by key. + +

      + +

      +
      • + +
    + +

    + Note that Couchbase Server provides atomicity, consistency, isolation, and durability + on operations that address a single document. + Thus, a single write is guaranteed either fully to succeed or to fail: no operation results in a partially + updated version of the document. Moreover, if an update operation imposes durability requirements — say, for + replicated or persisted durability — + the whole document is guaranteed to be durable. +

    + +
    + + + + +     diff --git a/content/understanding-couchbase/data/extended-attributes-fundamentals.dita b/content/understanding-couchbase/data/extended-attributes-fundamentals.dita new file mode 100644 index 0000000000..82fdee28d6 --- /dev/null +++ b/content/understanding-couchbase/data/extended-attributes-fundamentals.dita @@ -0,0 +1,116 @@ + + + + Extended Attributes + + + Couchbase Server permits the definition of extended attributes. These + allow developers to define application-specific metadata + visible only to those applications that request it or attempt to modify it. + This might be, for example, metadata specific to a programming framework that + should be hidden by default from other frameworks, or possibly from other versions of + the same framework. + + + + +
    + + Definition, Access, and Discovery + + +

    + The Couchbase SDK supports the creation and modification of extended attributes + by applications written in + Node.js, + Java, + .NET, + PHP, + Python, + Go, + and + C. + This is achieved by extensions to the + + Sub-document API. Extended attributes can be formed only as JSON, but can be + applied to both JSON and binary items. +

    + +

    + The creator-application can subsequently access and modify + the extended attributes it has created within a document. + However, Couchbase Server provides no facility for enumerating or sharing + knowledge of extended attributes: therefore, no application + has knowledge of the extended attributes within a + document other than their creator; unless such knowledge + is shared explicitly between applications by some mechanism external + to Couchbase Server. + +

    + +
    + +
    + + Implications for Sizing + + +

    + Within Couchbase Server, the maximum size for each document + is 20 megabytes (see the section + Size Limits, + on the page + Data). + Extended + attributes count against this size-limit: consequently, the size of a document may + reflect the presence of data inaccessible to some applications. +

    + +
    + +
    + + Virtual Extended Attributes + + +

    + A virtual extended attribute exposes additional information + about a document. Couchbase Server provides a single virtual + extended attribute, $document, which, when specified + as a search-path, returns metadata on the document. Ouput + might appear as follows: +

    + + DocumentFragment + { + id='hotel_10138', + cas=1504782798402879488, + mutationToken=null + } + [ + GET($document) + { + value= + { + "exptime":0, + "deleted":false, + "CAS":"0x14e20ffb82ec0000", + "seqno":"0x00000000000001fb", + "datatype":["json","xattr"], + "vbucket_uuid":"0x000060329233c341", + "value_bytes":1504, + "last_modified":"1504782798" + } + } + ] + +
    + + + + + + + + +     diff --git a/content/understanding-couchbase/data/images/document-data-model-flight-booking.png b/content/understanding-couchbase/data/images/document-data-model-flight-booking.png new file mode 100644 index 0000000000..4e6a50f482 Binary files /dev/null and b/content/understanding-couchbase/data/images/document-data-model-flight-booking.png differ diff --git a/content/understanding-couchbase/data/images/embed-vs-reference-ref-docs.png b/content/understanding-couchbase/data/images/embed-vs-reference-ref-docs.png new file mode 100644 index 0000000000..2d428e69ef Binary files /dev/null and b/content/understanding-couchbase/data/images/embed-vs-reference-ref-docs.png differ diff --git a/content/understanding-couchbase/data/images/item-maximum-sizes.png b/content/understanding-couchbase/data/images/item-maximum-sizes.png new file mode 100644 index 0000000000..c7b8e69e4c Binary files /dev/null and b/content/understanding-couchbase/data/images/item-maximum-sizes.png differ diff --git a/content/understanding-couchbase/data/images/jsonDataModel.png b/content/understanding-couchbase/data/images/jsonDataModel.png new file mode 100644 index 0000000000..cbdd4e287b Binary files /dev/null and b/content/understanding-couchbase/data/images/jsonDataModel.png differ diff --git a/content/understanding-couchbase/data/images/relational-model-flight-status.png b/content/understanding-couchbase/data/images/relational-model-flight-status.png new file mode 100644 index 0000000000..e9365793dd Binary files /dev/null and b/content/understanding-couchbase/data/images/relational-model-flight-status.png differ diff --git a/content/understanding-couchbase/data/images/relationalDataModel.png b/content/understanding-couchbase/data/images/relationalDataModel.png new file mode 100644 index 0000000000..befe356d7a Binary files /dev/null and b/content/understanding-couchbase/data/images/relationalDataModel.png differ diff --git a/content/understanding-couchbase/data/images/sub-doc-api-1.png b/content/understanding-couchbase/data/images/sub-doc-api-1.png new file mode 100644 index 0000000000..5cf0e3a754 Binary files /dev/null and b/content/understanding-couchbase/data/images/sub-doc-api-1.png differ diff --git a/content/understanding-couchbase/services-and-indexes/indexes/global-secondary-indexes.dita b/content/understanding-couchbase/services-and-indexes/indexes/global-secondary-indexes.dita new file mode 100644 index 0000000000..5eb21a8c6e --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/indexes/global-secondary-indexes.dita @@ -0,0 +1,71 @@ + + + + + Global Secondary Indexes + + + + A Global Secondary Index (GSI) supports queries made by the Query Service on attributes within + documents. Extensive filtering is supported. + + + +

    + Global Secondary Indexes provide the following: +

    + +

    +

    + +
      + +
    • + Predictable Performance: Key-based operations maintain predictable low-latency, even in + the presence of a large number of indexes. Index-maintenance is non-competitive with key-based + operations, even when data-mutation workloads are heavy. + +

      + +

      +
      • + +
    • + Low Latency Querying: Since the + Query Service + accesses GSIs, rather than data maintained in documents, it is not obliged to track how original and replica + vBuckets + are dispersed across nodes running the + Data Service. Moreover, + since the GSIs reside on specific cluster-nodes running the Index Service, the Query Service can access the GSIs + directly, without needing to use hashing routines. +

      + +

      +
      • + +
    • + Advanced Scaling: GSIs can be assigned independently to selected nodes, + without existing workloads being affected. + +

      + +

      +
      • + +
    + +
    + + Creating Global Secondary Indexes + +

    + Both Primary and Secondary indexes can be created by means of the N1QL query-language; using the CREATE INDEX + statement and the USING GSI clause. For more information, see + . +

    + +
    + + +     diff --git a/content/understanding-couchbase/services-and-indexes/indexes/images/query-exe-with-global-indexes.png b/content/understanding-couchbase/services-and-indexes/indexes/images/query-exe-with-global-indexes.png new file mode 100644 index 0000000000..55815e82e5 Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/indexes/images/query-exe-with-global-indexes.png differ diff --git a/content/understanding-couchbase/services-and-indexes/indexes/index-replication.dita b/content/understanding-couchbase/services-and-indexes/indexes/index-replication.dita new file mode 100644 index 0000000000..fbd6197648 --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/indexes/index-replication.dita @@ -0,0 +1,237 @@ + + + + + + Index Availability and Performance + + + + The Index Service ensures availability and performance through replication and + partitioning. The consistency of query-results can be controlled per query. + + + + +
    + + + Index Replication + + +

    + Secondary indexes can be replicated across cluster-nodes. This ensures: +

    + +

    +

    + +
      +
    • + Availability: If one Index-Service node is lost, the other continues to provide + access to replicated indexes. + +

      + +

      +
      • + +
    • + High Performance: If original and replica copies + are available, incoming queries are load-balanced across them. + +

      + +

      +
      • +
    + +

    + Index-replicas can be + created with the N1QL CREATE INDEX statement. Note that whenever + a given number of index-replicas is specified for creation, the number must be less than the number of cluster-nodes + currently running the + Index Service. + If it is not, the index-creation fails. Note also that if, following creation of the maximum number of copies, + the number of nodes running the Index Service + decreases, Couchbase Server progressively assigns replacement + index-replicas to any and all Index-Service nodes + subsequently be added to the cluster, until + the required number of index-replicas again exists for each replicated index. +

    + +

    + Index-replicas can be created as follows: +

    + +

    + +

    + +
      +
    • + Specifying, by means of the WITH clause, the destination nodes. In the following example, + an index with two replicas is created. The active index is on node1, and the replicas + are on node2 and node3: + + CREATE INDEX productName_index1 ON bucket_name(productName, ProductID) + WHERE type="product" USING GSI + WITH {"nodes":["node1:8091", "node2:8091", "node3:8091"]}; + +

      + +

      +
      • + +
    • + Specifying no destination nodes; but specifying instead, by means of the WITH clause + and the num_replica attribute, + only the number of replicas required. The replicas are automatically distributed across those nodes of the + cluster that are running the Index Service: the distribution-pattern is based on a projection of + optimal index-availability, given the number and disposition of Index-Service nodes across + defined server-groups. + +

      + In the following example, an index is created with two replicas, with no destination-nodes + specified: +

      + + CREATE INDEX productName_index1 ON bucket_name(productName, ProductID) + WHERE type="product" USING GSI + WITH {"num_replica": 2}; + +

      + Note that if + nodes and num_replica are both specified in the WITH + clause, the specified number of nodes must be one greater than num_replica. +

      + +

      + +

      +
      • + +
    • + Specifying a number of index-replicas to be created by the Index Service whenever CREATE INDEX is + invoked. The default is 0. If the default is changed to, say, + 2, creation of a single index is henceforth + accompanied by the creation of two replicas, which are automatically distributed across the nodes of the cluster + running the Index Service. No explicit specification within the CREATE INDEX statement is + required. + +

      + With credentials that provide appropriate authorization, this default can be changed; by means of + the curl command, as follows: +

      + + curl -u <username>:<password> <host>:9102/settings -d "{\"indexer.settings.num_replica\": <num_replicas>}" + +

      + Note, however, + that whenever explicit specification of replica-numbers is made + within the CREATE INDEX statement, this + explicit specification takes + precedence over any established default. +

      + +

      + +

      +
      • + +
    + +

    + For further information on using N1QL, see + Querying with N1QL. +

    + +
    + +
    + + + Index Partitioning + + +

    + As index-sizes and item-counts increase, individual indexes + may not fit on their assigned node. To remedy this, indexes can be partitioned (divided into two + or more segments) by means of the + WHERE clause to CREATE INDEX: the resulting partitions + can be distributed across multiple, individually specified Index-Service nodes, under + different index-names. +

    + +

    + In the following example, ranges are specified, to partition index-content + across two indexes: +

    + + CREATE INDEX productName_index1 ON bucket_name(productName, ProductID) + WHERE type="product" AND productName BETWEEN "A" AND "K" USING GSI + WITH {"nodes":["node1:8091"]}; + + CREATE INDEX productName_index2 ON bucket_name(productName, ProductID) + WHERE type="product" AND productName BETWEEN "K" AND "Z" USING GSI + WITH {"nodes":["node2:8091"]}; + +
    + +
    + + Index Consistency + + +

    + Whereas Couchbase Server handles data-mutations with full consistency — + all mutations to a given key are applied to the same vBucket, and become immediately + available — it maintains indexes with degrees of + eventual consistency, determined by means of the following query consistency-options, + specified per query: +

    + +
      + +
    • + not_bounded: Executes the query + immediately, without requiring any consistency for the query. If index-maintenance is + running behind, out-of-date results may be returned. + +

      +

      +
      • + +
    • + at_plus: Executes the query, requiring + indexes first to be updated to the timestamp of the last update. If + index-maintenance is running behind, the query waits for it to catch up. + +

      + +

      + +
      • + +
    • + request_plus: Executes the query, requiring the indexes first to be updated + to the timestamp of the current + query-request. If + index-maintenance is running behind, the query waits for it to catch up. + +

      +

      + +
      • + +
    + +

    + For N1QL, the default consistency is not_bounded. +

    + +
    + + + +     diff --git a/content/understanding-couchbase/services-and-indexes/indexes/indexes.dita b/content/understanding-couchbase/services-and-indexes/indexes/indexes.dita new file mode 100644 index 0000000000..a5cadb1c6c --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/indexes/indexes.dita @@ -0,0 +1,90 @@ + + + + + + Indexes + + + + Couchbase Server indexes enhance the performance of query and search operations. + + + + +
    + + Indexes + + +

    + The following forms of index are available: +

    + +
      + +
    • + Primary: Provided by the + Index Service, + this is + based on on the unique key of every item in a specified bucket. Every primary + index is maintained asynchronously. A primary index is intended to + be used for simple queries, which have no filters or predicates. For information on + creating a primary index, see + CREATE + PRIMARY INDEX. + +

      + +

      +
      • + +
    • + Secondary: Provided by the Index Service, this is based + on an attribute within a document. The value associated with the attribute + can be of any type: scalar, object, or array. + +

      + A Secondary Index is frequently referred to as a Global Secondary + Index, or GSI. This is the kind of index used most frequently + in Couchbase Server, for queries performed with the N1Ql query-language. + GSIs are also likely to be used extensively by the + Analytics Service. +

      + +

      + +

      + +
      • + +
    • + Full Text: Provided by the + Search Service, + this is + a specially purposed index, which contains targets + derived from the textual contents of documents within one or more + specified + buckets. Text-matches of different degrees of exactitude can be + searched for. Both input and target text-values can be purged of + irrelevant characters (such as punctuation marks or html tags). For information on + how to create Full Text Indexes, see + Creating Indexes. +
      • + +
    • + View: Supports Couchbase Views, with fields and information extracted from documents. + +

      + +

      +
      • + +
    + + +
    + + + +     diff --git a/content/understanding-couchbase/services-and-indexes/indexes/performance-consistency.dita b/content/understanding-couchbase/services-and-indexes/indexes/performance-consistency.dita new file mode 100644 index 0000000000..de2822fa77 --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/indexes/performance-consistency.dita @@ -0,0 +1,104 @@ + + + + Performance and Consistency + GSIs are designed to provide low latency scans for the query service. There are many + things that can be done to improve the performance of GSIs. Sometimes, however, you may wish to + compromise on performance in order to achieve better consistency for your queries. + +
    + Index Performance +

    GSIs provide a lower latency scan compared to view indexes due to its architecture. GSI, as + its name suggests is a global index that is independently placed within the cluster as + opposed to Views, which are local indexes placed aligned to the data distribution. + Query Execution with Global Indexes + +

    +

    Global secondary indexes can be placed on a single node. However as the number of index + scans, mutations, the total items indexed, and the total index size increases, it will be + necessary to scale the index. You can scale the index in 2 ways:

      +
    • Load balancing with GSI. You can create replicas of the same index to allow scans to + be routed to multiple nodes.
      • +
    • Partitioning with GSI. You can split the index into multiple partitions, to distribute + indexed items, index scan requests, and limit the mutations and size the single index + has to handle.
      • +
    In either approach, you also need to scale the index service to allocate more + computational resources to indexes.

    +
    + +
    Load Balancing with GSI
    +
    It is possible to create indexes with replicas distributed amongst multiple nodes. + Unlike replicas used in the data service, each replica index is considered to be + equivalent to the active index. This means that the query service automatically load + balances between the replica indexes, even when using the USE INDEX + statement or prepared statements.
    +
    See for further details on + index replication.
    +     + +
    Partitioning with GSI
    +
    Some of your indexes may no longer fit a single node as the number of mutations, index + size, and item count increases. You can partition a single index using a + WHERE clause with CREATE INDEX and by placing + partitions across multiple index service nodes under different index names.

    To + load-balance GSIs, you must manually specify the nodes on which index partitions + should be built on.

    You can also create indexes by partitioning your indexes as + shown in the following example using ranges: + CREATE INDEX productName_index1 ON bucket_name(productName, ProductID) + WHERE type="product" AND productName BETWEEN "A" AND "K" USING GSI + WITH {"nodes":["node1:8091"]}; + +CREATE INDEX productName_index2 ON bucket_name(productName, ProductID) + WHERE type="product" AND productName BETWEEN "K" AND "Z" USING GSI + WITH {"nodes":["node2:8091"]};

    +     + +
    Scaling the Index Service
    +
    Couchbase Server scales indexes independent of data and queries. With multidimensional + scaling, you can allocate separate hardware resources for separate services, and avoid + resource contention by performing queries, maintaining indexes, and writing data to + different nodes. If your application needs more indexing resources, you can either scale + out your infrastructure to add more index nodes, or scale up the index services to + handle more workload.
    +     +
    +
    +
    + Query and Index Consistency +

    In Couchbase Server, mutations to data in the data service is done with full consistency. + All mutations to a given key are done using the same vBucket on a node and are immediately + available to anyone reading the latest value for the given key. However, indexes are + maintained in an eventually consistent manner. This is true for all indexers (GSI, View, and + Spatial). At query time, you can specify a query consistency flag for each N1QL query + request, similar to the view API. The query consistency flag can be one of the following: +

      +
    • Not_bounded: This scan consistency flag executes the query + immediately without requiring any consistency for the query. If the index maintenance is + running behind, query may return out-of-date results. Not_bounded scan + consistency has the same characteristics as stale=ok in the view + API.
      • +
    • At_plus: This scan consistency flag executes the query but + require the indexes to be updated to the logical timestamp of the last update performed + by the application. For example, an application issuing the query may have done its last + update 10 ms ago. The sequence number of the update is retrieved from the mutation + response and is passed to the query request. This behavior achieves consistency, at + least or later than the moment of the most recent mutation from the application. If the + index maintenance is running behind, the query waits for the index to catch up to the + last update. At_plus scan consistency flag is not yet implemented by the + View API.
      • +
    • Request_plus: This scan_consistency flag + executes the query but require the indexes to be updated to the logical timestamp of the + query request. For example, an application issuing the query may have done its last + update 10 ms ago. An application issuing the query with request_plus + scan consistency flag takes the logical timestamp of the query request. This behavior + achieves consistency, at least or later than the moment of the request timestamp. If the + index maintenance is running behind the request timestamp, the query waits for the index + to catch up to the request timestamp, this can cause a delay in response to the query. + Request_plus scan consistency value has the same characteristics as + stale=false in the view API.
      • +
    For N1QL, the default consistency setting is not_bounded.

    +
    + +     diff --git a/content/understanding-couchbase/services-and-indexes/indexes/storage-modes.dita b/content/understanding-couchbase/services-and-indexes/indexes/storage-modes.dita new file mode 100644 index 0000000000..62bcb092a9 --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/indexes/storage-modes.dita @@ -0,0 +1,242 @@ + + + + + + Index Storage Settings + + + + A Secondary Index can be saved in either of two ways: standard + or memory-optmized. + + + +
    + + + Memory-Optimized Index Storage + + +

    + Memory-optimized index-storage allows high-speed maintenance and scanning; since the index is + kept fully in memory at all times. A snapshot of the index is maintained + on disk, to permit rapid recovery if node-failures are experienced. To be consistently beneficial, + memory-optimized index-storage + requires that all nodes running the Index Service have a memory quota sufficient + for the number and size of their resident indexes, and for the frequency with which the indexes will + be updated and scanned. +

    + +

    + Memory-optimized index-storage may be less suitable for nodes where memory is + constrained; since whenever + the Index Service memory-quota is exceeded, indexes on the node can neither be updated nor scanned. + In such circumstances, an error-notification is provided; and although the indexes remain in + ONLINE state, traffic is routed away from the node. + Before index-operations can resume, memory must be freed. +

    + +

    + In cases where recovery requires an Index-Service node to be restarted, the node's resident memory-optimized indexes + are rebuilt from the snapshots retained on disk. + Following the node's restart, these indexes remain in the BUILDING state + until all information has been read into memory: then, final updates are made with the indexes in + ONLINE state. + Note that once a rebuilt index is thus available, + queries with consistency=request_plus or + consistency=at_plus fail, if the specified timestamp exceeds the last + timestamp processed by given index. However, queries with + consistency=unbounded execute normally. For information on these settings, see + Index Availability and Performance. +

    + +

    + To rescue a node from an out-of-memory + situation, consider taking one or more of the following actions: +

    + +

    +

    + +
      + +
    • + Increase the index-memory quota, to give indexes additional memory + for request-processing. + +

      +

      +
      • + +
    • + Remove less important indexes from the node, to free up memory. + +

      +

      +
      • + +
    • + Remove buckets with indexes: removing a bucket automatically removes all the dependent + indexes. + +

      +

      +
      • + +
    • + Flush buckets that have indexes: flushing a bucket deletes all data in a bucket; and even if + there are pending updates not yet processed, flushing causes all indexes + to drop their own data. + +

      + Note that attempting to delete bucket-data selectively during an out-of-memory condition does + not succeed in decreasing memory-usage; since without memory, such requested deletions + cannot themselves be processed. +

      + +

      +

      +
      • + +
    + +
    + +
    + + + Standard Index Storage + + +

    + Standard is the default storage-setting for Secondary Indexes: the indexes are + saved on disk; in a disk-optimized format that uses both memory and disk for index-update + and scanning. +

    + + +

    + The performance of standard index storage + depends on overall I/O performance. Each index saved with the standard option + has two write modes: +

    + +

    + +

    + +
      + + +
    • + Circular Write Mode: Writes changes to the end of the index-file, until + the relative index fragmentation exceeds 65%. Block reuse is + then triggered: new data is written into stale blocks where possible, + rather than to the end of the file, so as to optimize I/O throughput. + Full compaction runs in accordance with the value of the + Circular write mode with day + time interval trigger setting: see + Index Fragmentation. + Note, however, that the index-fragmentation data-size is not significantly changed by compaction. + +

      +

      + +
      • + +
    • + Append-only Write Mode: Writes changes to the + end of the index-file, invalidating + existing pages within the index file, and requiring frequent, full compaction. + +

      +

      +
      • + +
    + +

    + By default, Couchbase Server uses Circular Write Mode for standard index storage. Append-only + Write Mode is provided for backwards compatibility with previous versions. These and other + storage-settings are described in detail in + Configuring Auto-Compaction. +

    + + +
    + + +
    + + + Changing Index-Storage Settings + + +

    + Settings + are established at cluster-initialization for + all indexes on the cluster, across all buckets. + Following + cluster-initialization, to change from one setting to the other, + all nodes running the Index Service must be removed. If + the cluster is single-node, uninstall and reinstall Couchbase + Server. If the cluster is multi-node, and only some of the nodes host + the Index Service, proceed as follows: +

    + +

    + +

    + +
      +
    1. + Identify the nodes running the Index Service. + +

      +

      +
      2. + +
    2. + Remove each of the nodes running the Index Service. + Note that as Index-Service nodes are removed, so are the indexes they contain; and in consequence, + any ongoing queries fail. + + +

      +

      +
      3. + +
    3. + Perform a rebalance. + +

      +

      +
      4. + +
    4. + Change the Index-Storage Settings for the + cluster. + +

      +

      +
      5. + +
    5. + Add new Index-Service nodes, and confirm the revised storage mode. + +

      + +

      +
      6. + +
    + +

    + For information on adding and removing nodes, and on rebalancing a cluster, see + Cluster Operations. +

    + + +
    + +     diff --git a/content/understanding-couchbase/services-and-indexes/services-and-indexes.dita b/content/understanding-couchbase/services-and-indexes/services-and-indexes.dita new file mode 100644 index 0000000000..7d0af3d19f --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/services-and-indexes.dita @@ -0,0 +1,61 @@ + + + + + + Services and Indexes + + + + Couchbase Services support access to and maintenance of data. + Indexes enhance the performance of query and search operations. + + + + +
    + + + Services + + +

    + Services are configured and deployed by the Full Administrator who initializes + Couchbase Server on one or more nodes. The standard configuration-sequence allows + a subset of services to be selected per node, with an individual memory-allocation for each. Each service supports + a particular form of data-access. Services not required need not be deployed. + Services intended to support a heavy workload can be deployed across + multiple cluster-nodes, to ensure optimal performance and resource-availability. +

    + +
    + +
    + + Indexes + + +

    + Indexes are used by certain services, such as + Query, Analytics, and Search, as targets + for search-routines. + Each index makes a predefined + subset of bucket-data available for the search. +

    + +

    + The Query and Analytics + services both rely on indexes provided by the Index service. The + Search service provides its own indexes, internally. +

    + +

    + Indexes, when well-designed, provide significant enhancements to the performance of + search-operations. +

    + +
    + + + +     diff --git a/content/understanding-couchbase/services-and-indexes/services/analytics-service.dita b/content/understanding-couchbase/services-and-indexes/services/analytics-service.dita new file mode 100644 index 0000000000..0a388ba315 --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/services/analytics-service.dita @@ -0,0 +1,69 @@ + + + + + Analytics Service + + + The Analytics Service provides a parallel data-management capability; allowing + the running of complex analytical queries. + + + + +
    + + + Understanding Analytics + + +

    + The Analytics Service supports large join, set, aggregation, and + grouping operations: these activities are anticipated often to employ large amounts + of data, and therefore to be highly + consumptive of processor, networking, and memory resources; highly demanding in terms of + cross-node coordination; and of extensive duration. +

    + +

    + Analytic queries can be either predetermined or ad hoc; + predetermined queries often providing greater efficiency. The Analytics Service + further enhances efficiency by supporting parallel query-processing and bulk data-handling; + and by allowing analytic queries to be run on indexes. +

    + +
    + +
    + + + Deploying the Analytics Service + + +

    + The Analytics Service relies on the + Data Service, + which must therefore be running on at least + one of the cluster nodes. For the Analytics Services to use + indexes, the + Index Service + must also be running on at least one of the cluster nodes. +

    + +

    + Due to the large scale and duration of operations it is likely to perform, the Analytics Service should be + run alone, on its own cluster node, with no other Couchbase Service running on that node. +

    + +

    + For the practical steps required to initialize or join a cluster, and to deploy services, see + Initialize the Cluster. + For information on how to run analytic queries, see the + Introduction to + Couchbase Analytics. +

    + +
    + + +     diff --git a/content/understanding-couchbase/services-and-indexes/services/data-service.dita b/content/understanding-couchbase/services-and-indexes/services/data-service.dita new file mode 100644 index 0000000000..5cd8a1cc69 --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/services/data-service.dita @@ -0,0 +1,195 @@ + + + + + + Data Service + + + + The Data Service provides access to data. + + + + +
    + + Understanding the Data Service + + +

    + The Data Service is the most fundamental of all Couchbase services, providing access + to data in memory and on disk. The memory allocation for the Data Service is configurable. The + Data Service must run on at least one node of every cluster. +

    + +

    + The architecture of the Data Service is illustrated as follows: +

    + +

    + +

    + +

    + The principal components are: +

    + +

    + +

    + +
      +
    • + Dispatcher: Manages networking, by handling each Request for data, and providing the + Response. Also streams data to other nodes within the cluster, and to other clusters, by means + of the DCP protocol; and handles Authentication. + +

      + For information on DCP Streaming, see + Clusters and Availability. + For information on Authentication, see + Authentication. + For information on ports used by the dispatcher, see + Network Configuration. +

      + +

      +

      +
      • + +
    • + KV Engine: A collection of facilities that is provided for each bucket on the cluster. These are: + +

      +

      + +
        +
      • + Managed Cache: The memory allocated for the bucket, according to an established quota. + This contains Partition Hash Tables, whereby + the location of bucket-items, in memory and on disk, on different nodes across the cluster, is recorded. When + written, items enter the cache, and subsequently are placed onto a replication queue, so as to be replicated + to one or more other nodes; and (in the case of items for Couchbase buckets) onto a disk queue, so as to + be written to disk. + +

        + For information on memory quotas, see + Memory. + For information on how items from a bucket are assigned to vBuckets across a cluster, see + vBuckets. + For illustrations of how items are written and updated for Couchbase buckets, see + Memory and Storage. +

        +

        + +

        +
        • + +
      • + Checkpoint Manager: Keeps track of item-changes, using data structures named checkpoints. Changes + already made to items in memory, but not yet placed on the replication and disk queues, are recorded. +

        + +

        +
        • + +
      • + Item Pager: Ejects from memory items that have not recently been used, in order to free up space, as required. + For further information, see + Memory. +

        + +

        +
        • + +
      • + Flusher: Deletes every item in the bucket. For information on how to activate, see + Flush a Bucket. +

        + +

        +
        • + +
      • + Expiry Pager: Scans for items that have expired, and erases them from memory and disk; after + which, a tombstone remains for a default period of 3 days. The expiry pager runs + every 60 minutes by default: for information on changing the interval, see cbepctl + set flush_param. + For more information on item-deletion and tombstones, see + Expiration. +

        + +

        +
        • + +
      • + Batch Reader: Enhances performance by combining changes made to multiple items into batches, + which are placed on the disk queue, to be written to disk. +

        + +

        +
        • +
      + +

      + +

      +
      • + +
    • + Scheduler: A pool of threads, mainly purposes for handling I/O. The threads are divided into four kinds, which + run independently of and without effect on one another: + +

      +

      + +
        +
      • + Non IO: Tasks private to the scheduler that do not require disk-access; including + connection-notification, checkpoint removal, and hash-table resizing. + +

        +

        +
        • + +
      • + Aux IO: Fetch, scan, and backfill tasks. + +

        +

        +
        • + +
      • + Reader IO: Threads that read information from disk. + +

        +

        +
        • + +
      • + Writer IO: Threads that write information to disk. + +

        +

        +
        • + +
      + +

      + For more information on the Data Service' threading model, see + Storage. +

      + +
      • + +
    + +
    + + + + + + +     diff --git a/content/understanding-couchbase/services-and-indexes/services/eventing-service.dita b/content/understanding-couchbase/services-and-indexes/services/eventing-service.dita new file mode 100644 index 0000000000..6433c9e3e4 --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/services/eventing-service.dita @@ -0,0 +1,42 @@ + + + + + + Eventing Service + + + + The Eventing Service provides near real-time handling of changes to data; whereby + code is executed either in response to mutations, or as scheduled by timers. + + + + +
    + + + Understanding Eventing + + +

    + The Eventing Service allows functions to be written, saved, and triggered in response + to events. Events include changes made to specified items, and the arrival of scheduled + points-in-time. +

    + +

    + The Eventing Service depends on the + Data Service, which must + therefore be running on at least one cluster node. +

    + +

    + For information on using the Eventing Service, see + Eventing Service: Fundamentals. +

    + +
    + + +     diff --git a/content/understanding-couchbase/services-and-indexes/services/images/cbClusterWithServicesDevelopment.png b/content/understanding-couchbase/services-and-indexes/services/images/cbClusterWithServicesDevelopment.png new file mode 100644 index 0000000000..bf7e521125 Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/cbClusterWithServicesDevelopment.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/images/cbClusterWithServicesProduction.png b/content/understanding-couchbase/services-and-indexes/services/images/cbClusterWithServicesProduction.png new file mode 100644 index 0000000000..32125bc234 Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/cbClusterWithServicesProduction.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/images/dataServiceArchitecture.png b/content/understanding-couchbase/services-and-indexes/services/images/dataServiceArchitecture.png new file mode 100644 index 0000000000..aaba4254b4 Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/dataServiceArchitecture.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/images/db-engine-architecture.png b/content/understanding-couchbase/services-and-indexes/services/images/db-engine-architecture.png new file mode 100644 index 0000000000..a83fd53d1e Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/db-engine-architecture.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/images/indexServiceArchitecture.png b/content/understanding-couchbase/services-and-indexes/services/images/indexServiceArchitecture.png new file mode 100644 index 0000000000..68fda445c1 Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/indexServiceArchitecture.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/images/n1ql-query-engine-components.png b/content/understanding-couchbase/services-and-indexes/services/images/n1ql-query-engine-components.png new file mode 100644 index 0000000000..da4efb5385 Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/n1ql-query-engine-components.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/images/n1ql-query-execution-flow.png b/content/understanding-couchbase/services-and-indexes/services/images/n1ql-query-execution-flow.png new file mode 100644 index 0000000000..058fd14122 Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/n1ql-query-execution-flow.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/images/querySequence.png b/content/understanding-couchbase/services-and-indexes/services/images/querySequence.png new file mode 100644 index 0000000000..6cdcd0269c Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/querySequence.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/images/queryServiceArchitecture.png b/content/understanding-couchbase/services-and-indexes/services/images/queryServiceArchitecture.png new file mode 100644 index 0000000000..771766d517 Binary files /dev/null and b/content/understanding-couchbase/services-and-indexes/services/images/queryServiceArchitecture.png differ diff --git a/content/understanding-couchbase/services-and-indexes/services/index-service.dita b/content/understanding-couchbase/services-and-indexes/services/index-service.dita new file mode 100644 index 0000000000..93cb8c5367 --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/services/index-service.dita @@ -0,0 +1,174 @@ + + + + + Index Service + + + The Index Service supports the creation of primary and secondary indexes + on items stored within Couchbase Server. + + + + +
    + + + Index Service Architecture + + +

    + Components essential for the Index Service reside not only on + each node to which the Index Service is assigned, but also on each node to which the + Data Service is assigned, as shown by the following illustration: +

    + +

    + +

    + +

    + The illustration depicts the following: +

    + +

    +

    + +
      +
    • + Data Service: Uses the DCP protocol to stream data-mutations + to the Projector and Router + process, which runs as part of the Index Service, on each Data Service node. +

      + +

      +
      • + +
    • + Projector and Router: Provides data to the Index Service, according to the + index-definitions provided by the Index Service Supervisor. + +

      + When the Projector and Router starts running on the Data Service-node, + the Data Service streams to the Projector and Router copies of all + mutations that occur to bucket-items. Prior to the creation of any indexes, the + Projector and Router takes no action. +

      + +

      + When an index is first created, the Index Service + Supervisor contacts the Projector and + Router; and passes to it the corresponding index-definitions. The Projector and Router + duly contacts the Data Service, and + extracts data from the fields specified by the index-definitions. It then sends the + data to the Supervisor, so that the index can be populated. +

      + +

      + Subsequently, the Projector and Router continuously examines the stream of mutations provided + by the Data Service. When this includes a mutation to an indexed field, the mutated data is + passed by the Projector and Router to the Supervisor, and the index thereby + updated. +

      + +

      + +

      +
      • + +
    • + Supervisor: The main program of the Index Service; which passes index-definitions + to the Projector and Router, creates and stores indexes, and + handles mutations sent from the Projector and Router. + +

      + +

      +
      • + +
    • + Query Service: Passes to the Supervisor clients' create-requests and queries, and handles + the responses. +

      + +

      +
      • + +
    + +

    +

    + + +
    + +
    + + Saving Indexes + + +

    + By default, an index is saved on the node on which it is created. Each index is + created on one bucket only; but multiple indexes may be created on a single + bucket. +

    + +

    + The Index Service can be configured to use either standard or memory-optimized + storage: +

    + +

    +

    + +
      +
    • + Standard: The supervisor process persists to disk all changes made to + individual indexes. Each index gets a dedicated file. Writes + can either be append-only or write with circular reuse, + depending on the write mode selected for global secondary indexes. +

      + +

      +
      • + +
    • + Memory-Optimized: Indexes are saved in-memory. This provides + increased efficiency for maintenance, scanning, and mutation. + efficient index maintenance and index scans. + A skiplist (rather than a conventional B-tree) structure + is used; optimizing memory consumption. Lock-free index-processing + enhances concurrency. +

      + +

      +
      • + + +
    + +

    + For more information, see + Index Storage Settings. +

    + + +
    + +
    + + Using Global Secondary Indexes + + +

    + Secondary Indexes, often referred to as Global Secondary Indexes or GSIs, constitute the principal + means of indexing documents to be accessed by the Query Service. Secondary Indexes can be created on specified + fields; placed on specific cluster-nodes; and replicated. + For information, see + Global Secondary Indexes +

    + +
    + + +     diff --git a/content/understanding-couchbase/services-and-indexes/services/query-service.dita b/content/understanding-couchbase/services-and-indexes/services/query-service.dita new file mode 100644 index 0000000000..f04f18eb2c --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/services/query-service.dita @@ -0,0 +1,111 @@ + + + + + Query Service + + + The Query Service supports the querying of data by means of the N1QL query language. + The Query Service depends on both the Index Service and the Data Service. + + + + +
    + + + Query Service Architecture + + +

    + The architecture of the Query Service is shown by the following illustration: +

    + +

    + +

    + +

    + The principal components are: +

    + +

    +

    + +
      +
    • + Listeners: Concurrent query requests are received on ports 8093 and 18093 (for more information + on port-allocation, see + Network Configuration). +

      + +

      +
      • + +
    • + Query Processor: Responsible for applying the Parser to incoming queries, in order to + determine whether each is a valid statement. Also employs the Optimizer, which evaluates available + execution paths, so determining the path of lowest latency; generates a query-execution plan that uses + the lowest-latency path; and assembles the plan into a series of operators. The Execution Engine + receives the operators, and executes them — in parallel where possible. +

      + +

      +
      • + +
    • + Data Stores: Provides access to various data-sources. The Couchbase Server store is used to + access the data and indexes on Couchbase Server, and to handle authentication. Other data stores are also + included, such as the store for the local filesystem. +

      + +

      +
      • + +
    + +
    + +
    + + Query Execution + + +

    + The sequence whereby queries are executed is shown below: +

    + +

    + +

    + +

    + The client's N1QL query is shown entering the Query Service at the left-hand side. The Query Processor performs + its Parse routine, to validate the submitted statement, then creates the execution Plan. Scan operations + are then performed on the relevant index, by accessing the Index Service. Next, Fetch + operations are performed by accessing the Data Service, and the data duly returned is used in Join operations. + The Query Service continues by performing additional processing, which includes Filter, Aggregate, and + Sort operations. Note the degree of parallelism with which operations are frequently performed, represented + by the vertically aligned groups of right-pointing arrows. +

    + +
    + +
    + + + Using N1QL + + +

    + The Query Service supports queries made in the N1QL query language. As well as providing a rich + variety of query-options, N1QL allows statements to be prepared, so that the parsing + and compiling of plans is completed prior to execution; and permits consistency-levels to be + configured. For detailed information, see + Querying with N1QL. +

    + +
    + + +     diff --git a/content/understanding-couchbase/services-and-indexes/services/search-service.dita b/content/understanding-couchbase/services-and-indexes/services/search-service.dita new file mode 100644 index 0000000000..66f01b26ed --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/services/search-service.dita @@ -0,0 +1,75 @@ + + + + + Search Service + + + The Search Service supports the creation of specially purposes indexes for + Full Text Search. + + + + +
    + + + Understanding Search + + +

    + Search represents Full Text search. The indexes that it creates and + uses are entirely separate from and different to those of the Index Service. + Full Text Search provides extensive capabilities for natural-language + querying. These include: +

    + +
      + +
    • + Language-aware searching; allowing users to search for, say, the word + beauties, and additionally obtain results + for beauty and beautiful. + +

      + +

      +
      • + +
    • + Scoring of results, according to relevancy; allowing + users to obtain result-sets that only contain documents awarded the highest + scores. This keeps result-sets manageably small, even when the total number of documents + returned is extremely large. + +

      + +

      +
      • + +
    • + Fast indexes, which support a wide range of possible text-searches. + +

      + +

      +
      • + + + +
    + +

    + The Search Service requires the Data Service, but does not require, and has no relation to + the Index Service or Query Service. +

    + +

    + For extensive details on how to use the service, see + Full Text Search: Fundamentals. +

    + +
    + + +     diff --git a/content/understanding-couchbase/services-and-indexes/services/services.dita b/content/understanding-couchbase/services-and-indexes/services/services.dita new file mode 100644 index 0000000000..9db787a2c8 --- /dev/null +++ b/content/understanding-couchbase/services-and-indexes/services/services.dita @@ -0,0 +1,158 @@ + + + + + + Services + + + + Couchbase Services support access to and maintenance of data. + + + + +
    + + + Services + + +

    + Couchbase Server provides the following services: +

    + +

    + +

    + +
      + +
    • + Data: Supports the storing, setting, and retrieving of data-items, specified by + key. +

      + +

      +
      • + +
    • + Query: Parses queries specified in the N1QL query-language, executes + the queries, and returns results. The Query Service interacts with both the + Data and Index services. +

      + +

      +
      • + +
    • + Index: Creates indexes, for use by the Query and Analytics + services. +

      + +

      +
      • + +
    • + Search: Create indexes specially purposed for Full Text Search. This + supports language-aware searching; allowing users to search for, + say, the word beauties, and additionally obtain + results for beauty and beautiful. +

      + +

      +
      • + +
    • + Analytics: Supports join, set, + aggregation, and grouping operations; which are expected to + be large, long-running, and highly consumptive of memory and CPU resources. +

      + +

      +
      • + +
    • + Eventing: Supports near real-time handling of changes to + data: code can be executed both in response to document-mutations, + and as scheduled by timers. +

      + +

      +
      • + +
    + +
    + +
    + + Setting Up Services + + +

    + Services are set up on a per node basis. Each node can run at most one instance of + a service. The Data Service must run on at least one node. Some services are + interdependent, and therefore require at least one instance of each of their dependencies to be running on + the cluster (for example, the Query Service depends on the + Index Service and on the Data Service). +

    + +

    + When the first node in a cluster is initialized, the services assigned to it become + the default assignment for each other node subsequently to be added to the cluster. + However, this default can be departed from, node by node; with one or more services omitted from the + default, and one or more added. +

    + +

    + When first allocated to a node, a service requires the assignment of a specific memory quota, + which becomes standard for that service in each of its instances across the cluster. (The + exception to this is the Query Service, which never requires a memory quota.) +

    + +

    + Service-allocation should be designed based on workload-analysis: if a particular service is + expected to handle a heavy workload, it should be allocated with a larger memory quota, and + potentially as the only service on the node. Alternatively, if a cluster is to be used for + development purposes only, it may be convenient to allocate services in the quickest and + most convenient way, with some quotas being equal. +

    + +

    + For example, the following illustration shows how four services — Data, Index, Query, and Search — + might be allocated evenly across the five nodes of a development cluster: +

    + +

    + +

    + +

    + This configuration might provide perfectly acceptable performance for each of the four services, in the context + of development and testing. However, if a large amount of data needed, in production, to be intensively indexed, + and addressed by means of Query and Search, the following configuration would be more efficient: +

    + +

    + +

    + +

    + In this revised configuration, the Data Service is the only service to run on two of the nodes; the Index Service the only + service on two futher nodes; and the Query and Search Services share the fifth and final node. +

    + +

    + For a more detailed explanation of service memory quotas, see + Memory. + For information on the practical steps required to initialize a cluster, including the allocation + of services to nodes, see + Initialize the Cluster. +

    + +
    + + + +     diff --git a/content/understanding-couchbase/understanding-couchbase.dita b/content/understanding-couchbase/understanding-couchbase.dita new file mode 100644 index 0000000000..c9e9ba008f --- /dev/null +++ b/content/understanding-couchbase/understanding-couchbase.dita @@ -0,0 +1,593 @@ + + + + + + Overview + + + + An overview of Couchbase Server's core features. + + + + +
    + + + Core Features + + +

    + This section provides descriptions of Couchbase Server's core features. The descriptions are + organized as follows: +

    +

    + +

    + +
      +
    • +

      + Data: Couchbase Server stores data as items. + Each item consists of a key, by which + the item is referenced; and an associated value, which must be either binary + or a JSON document. +

      + +

      + See + Data for information. +

      + +

      + +

      +
      • + +
    • +

      + Buckets, Memory, and Storage: Items are stored in named Buckets; + being kept only in memory, others both in memory and on disk. +

      + +

      + See + Buckets, Memory, and Storage + for information. +

      + +

      +

      +
      • + +
    • +

      + Services and Indexes: Services can be deployed to + support different forms of data-access: + for example, the Data Service allows items to be retrieved by key; + while the Query Service allows them to be retrieved by means of queries, + designed in the N1QL query-language. Individual services + can be configured to run across multiple cluster-nodes, allowing + high-priority workloads to be distributed and scaled appropriately. Indexes support + services, by enabling high-performance access to data. +

      + +

      + See + Services and Indexes for + information. +

      + +

      +

      +
      • + +
    • +

      + Clusters and Availability: A single node running Couchbase Server is considered + a cluster of one node. As successive nodes are + initialized, each can be configured to + join the existing cluster. +

      + +

      + Across the nodes of each cluster, Couchbase data is evenly distributed and replicated: nodes can + be removed, and node-failure handled, without data-loss. Data can be selected for + replication across clusters residing in different data centers, to ensure high availability. +

      + +

      + See + Clusters and Availability + for information. +

      + +

      +

      +
      • + +
    + +
    + +
    + + + Couchbase Glossary + + +

    + The following glossary introduces the principal terms used in descriptions of Couchbase + Server-technology. Use the links to locate the full descriptions of each. +

    + +

    + +

    + +
      + +
    • + Server: An instance + of Couchbase Server — an open source, distributed, NoSQL document-oriented engagement database, + specialized to provide low-latency data management for large-scale interactive web, mobile, and other + applications. Each instance runs on its on physical or virtual machine. + +

      + +

      +
      • + +
    • + Cluster: One + or more instances of Couchbase Server, + each running on an independent node; but cooperating with any and all others, so as to form a unified system; + whereby resources are + shared, and a single interface provided for data-access and management. + +

      +

      +
      • + +
    • + Bucket: A logical, user-named entity that + groups items; allowing them to be accessed, indexed, replicated, and access-controlled. There are three types: + +

      + +

      + +
        +
      • + Couchbase: Retains data both in memory and on disk. +

        + +

        +
        • + +
      • + Ephemeral: Retains data in memory only. +

        + +

        +
        • + +
      • + Memcached: Designed to be used in the context of other database platforms, such as ones + employing relational database technology, in order to provide a managed memory-cache for frequently-used data. + +

        + +

        +
        • + +
      + +
      • + +
    • + Memory: An automatically managed caching + layer, supporting high-speed data-access. + +

      +

      +
      • + +
    • + Storage: The + persistent retention of items on disk, in compressed form, with high-speed threaded access. + +

      +

      +
      • + + + +
    • + Data: Items, each of which consists of a key + by which the item is referenced; and an associated value, which must + be either binary or a JSON document. + +

      +

      + +
        + +
      • + Access: The creation, update, and deletion of items, as supported by the + Couchbase Web Console UI and the + Couchbase SDK. + +

        + +

        +
        • + +
      • + Model: A lightweight, flexible + schema; which can be progressively evolved by applications over time, and allows information + to be stored in the form of items. + +

        + +

        +
        • + +
      + +
      • + +
    • + Node: A computer + (potentially, a virtual machine) running an instance + of Couchbase Server. + +

      +

      + +
        + +
      • + Addition: The + ability to add a Couchbase + Cluster of one node to another existing cluster, + so that a single, combined cluster is produced. Following addition, rebalance + ensures that data and indexes are optimally distributed across all available nodes. + +

        + +

        + +
        • + +
      • + Failover: The ability + to allow healthy nodes to continue functioning as a cluster, potentially without data-loss, + when one node has gone offline. Rebalance + ensures that data and indexes are optimally distributed across all available nodes. Failover can be + automated. + +

        +

        + +
        • + +
      • + Removal: The + ability to remove a node + from a cluster. Following removal, rebalance + ensures that data and indexes are optimally distributed across all available nodes. + +

        +

        + +
        • + +
      +
      • + +
    • + Rebalance: The process of + redistributing data and indexes optimally among + the available nodes of a cluster. This should be performed whenever a + cluster-configuration has changed. + +

      +

      +
      • + + + +
    • + Availability: + The preservation of data from system-failure, by the following means: + +

      +

      + +
        + +
      • + Backup and Restore: The storing in archive-repositories + of the current state of data, indexes, + and bucket configurations; and the restoration of such state to a running + cluster. + +

        + +

        +
        • + +
      • + Cross Datacenter Replication (XDCR): + The replication of data between clusters, + to ensure the least chance of data-loss + in the event of data-center failure, and to provide high-performance data-access + for globally distributed applications. +

        + +

        +
        • + +
      • + Data Recovery: The + restoration of current data to a node that is recovered from failure: either + by updating data still held locally, or by substituting current data from other + nodes. +

        + +

        +
        • + +
      • + Intra-Cluster Replication: The maintenance and continuous update of data-copies, + distributed across the nodes of a cluster, to ensure the least chance of data-loss in the event of + single-node failure. + +

        + +

        +
        • + +
      + + +
      • + + +
    • + Deployment: The installation of a + single instance of Couchbase Server, subsequent to the + appropriate resourcing and configuration of an underlying platform. + +

      +

      + +
        + + +
      • + Cloud: Couchbase Server + runs on a pre-established cloud-configuration. +

        + +

        +
        • + +
      • + Container: Couchbase + Server runs within a software container or virtual machine. +

        + +

        +
        • + +
      • + Native: Couchbase Server runs on + an individual, physical machine. +

        + +

        +
        • + +
      +
      • + +
    • + Initialization: The configuration of a + new instance of Couchbase Server, either as the first + node in a new cluster, or as an additional node for an existing cluster. + +

      +

      +
      • + + + + +
    • + Security: Couchbase-Server Authentication, Authorization, Auditing, and Encryption. + +

      +

      + +
        +
      • + Authentication: To + access Couchbase Server, administrators and applications must be authenticated. + Authentication is a process for identifying a user who is attempting to access + a system. Authentication can be attempted by passing credentials, or by means + of certificates. + +

        + +

        +
        • + +
      • + Authorization: Role-Based + Access Control (RBAC), whereby access-privileges + are assigned to fixed roles that are themselves assigned to administrators and + applications. +

        + +

        +
        • + +
      • + Auditing: The detailed, automated + recording of actions performed on Couchbase + Server, allowing administrative review; in order to ensure that system-management + tasks are being appropriately performed. +

        + +

        +
        • + +
      • + Encryption: SSL/TLS, + used to encrypt data passed over the wire. + +

        + +

        +
        • + +
      +
      • + +
    • + Services: Couchbase + Server-facilities that support different forms of data-access: + +

      + +

      + +
        + +
      • + Analytics: + Supports join, set, aggregation, and grouping operations that are expected to + be large, long-running, and highly consumptive of memory and CPU resources. +

        + +

        +
        • + +
      • + Data: Supports + the storing, setting, and retrieving of data-items, specified by key. +

        + +

        +
        • + +
      • + Eventing: Supports + near real-time handling of changes to data: code can be executed both + in response to document-mutations, and as scheduled by timers. +

        + +

        + +
        • + + +
      • + Index: + Creates indexes, for use by the Query and Analytics services. +

        + +

        +
        • + +
      • + Query: Parses + input specified in the N1QL query-language, executes queries, and + returns results. The Query Service interacts with both the Data and Index + services. +

        + +

        +
        • + +
      • + Search: Creates indexes + specially purposed for Full Text Search. This supports + language-aware searching; allowing users to search for, say, the word beauties, + and additionally obtain results for beauty and beautiful. +

        + +

        +
        • + +
      +
      • + +
    • + Scaling: The optional + allocation of services to cluster-nodes in accordance + with workload-requirements. For example, if a particular service is expected to + handle a heavy workload, it can be allocated a large memory quota, and + might be deployed as the only service on its node, to ensure optimal + availability of CPU cycles. + +

      +

      +
      • + +
    • + Tools: Provided by Couchbase Server + to support cluster-management: + +

      + +

      + +
        + +
      • + CLI: Command-line-based management. +

        +

        +
        • + +
      • + Couchbase Web Console: UI-based management. + +

        +

        +
        • + +
      • + REST API: RESTful management. + Note that the REST API, as well as being directly + available to the administrator, also underlies the features of the + Couchbase Web Console and CLI. + +

        +

        +
        • + +
      +
      • + +
    • + SDK: Libraries that support + cluster-access for applications written in Java, .NET, C, Go, + PHP, Python, and NodeJs. + +

      +

      +
      • + +
    + + +
    + + + +     diff --git a/content/understanding-couchbase/views/data-json.dita b/content/understanding-couchbase/views/data-json.dita new file mode 100644 index 0000000000..781fc77d92 --- /dev/null +++ b/content/understanding-couchbase/views/data-json.dita @@ -0,0 +1,139 @@ + + + + JSON Data + JSON is a lightweight, easily parsed, cross-platform data representation format. + + + +

    There are a multitude of libraries and tools + designed to help developers work efficiently with data represented in JSON format, on every + platform and every conceivable language and application framework, including, of course, + most web browsers.

    +

    JSON supports the same basic types as supported by JavaScript, including:

    + +
      +
    • Number (either integer or floating-point)

      JavaScript supports a maximum numerical value + of 253. If larger numbers within the client library environment are used (for + example, 64-bit numbers), store the value as a string.

      • +
    • String +

      This should be enclosed by double-quotes and supports Unicode characters and + backslash escaping. For example: + "A String"

      • +
    • Boolean

      A true or false value. These strings are used + directly. For example: { "value": true}

      • +
    • Array +

      List of values enclosed in square brackets. For example: + ["one", "two", "three"]

      • +
    • Object +

      Set of key/value pairs (i.e. an associative array, or hash). The key must + be a string, but the value can be any of the supported JSON values. For example: + { + "servings" : 4, + "subtitle" : "Easy to make in advance, and then cook when ready", + "cooktime" : 60, + "title" : "Chicken Coriander" +} +

      • +
    + + + + +
    Document metadata +

    During view processing, metadata about individual + documents is exposed through a separate JSON object, meta, that can be + optionally defined as the second argument to the map(). This metadata can + be used to further identify and qualify the document being processed.

    +

    The + meta structure contains the following fields and associated + information:

    + + +
      +
    • id +

      The ID or key of the stored data object. This is the same as the key used when writing + the object to the Couchbase database.

      • +
    • rev +

      An internal revision ID used internally to track the current revision of the + information. The information contained within this field is not consistent or trackable and + should not be used in client applications.

      • +
    • type +

      The type of the data that has been stored. A valid JSON document will have the type + json. Documents identified as binary data will have the type + base64.

      • +
    • flags +

      The numerical value of the flags set when the data was stored. The availability and + value of the flags is dependent on the client library you are using to store your data. + Internally the flags are stored as a 32-bit integer.

      • +
    • expiration +

      The expiration value for the stored object. The stored expiration time is always + stored as an absolute Unix epoch time value.

      These additional fields are only exposed + when processing the documents within the view server. These fields are not returned when you + access the object through the Memcached/Couchbase protocol as part of the + document.

      • +
    +
    + + +
    Retrieving JSON data + +

    Information identified as JSON data may not be returned in a format identical to that stored. + The information will be semantically identical, in that the same fields, data and structure as + submitted will be returned. Metadata information about the document is presented in a separate + structure available during view processing.

    +

    The white space, field ordering may differ from the submitted version of the JSON + document.

    +

    For example, the JSON document below, stored using the key mykey :

    + + +{ + "title" : "Fish Stew", + "servings" : 4, + "subtitle" : "Delicious with fresh bread" +} + + +

    May be returned within the view processor as:

    + +{ + "servings": 4, + "subtitle": "Delicious with fresh bread", + "title": "Fish Stew" +} + +
    + + +
    Document storage and indexing sequence

    The method of storage of + information into the Couchbase Server affects how and when the indexing information is + built, and when data written to the cluster is incorporated into the indexes. In addition, + the indexing of data is also affected by the view system and the settings used when the view + is accessed.

    + +

    The basic storage and indexing sequence is:

    + +
      +
    1. A document is stored within the cluster. Initially the document is stored only in RAM.
      2. +
    2. The document is communicated to the indexer through replication to be indexed by views.
      3. +
    + +

    This sequence means that the view results are eventually consistent with what is stored in + memory based on the latency in replication of the change to the indexer. It is possible to + write a document to the cluster and access the index without the newly written document + appearing in the generated view.

    + + +

    Conversely, documents that are stored with an expiry may continue to be included within the + view until the document has been removed from the database by the expiry pager.

    + +

    Couchbase Server supports the Observe command, which enables the current state of a document and + whether the document has been replicated to the indexer or whether it has been considered for inclusion in an index. +

    +

    When accessing a view, the contents of the view are asynchronous to the stored documents. + In addition, the creation and updating of the view is subject to the parameter. + This controls how and when the view is updated when the view content is queried. +

    +
    +     +     diff --git a/content/understanding-couchbase/views/data-nonjson.dita b/content/understanding-couchbase/views/data-nonjson.dita new file mode 100644 index 0000000000..d9deae3840 --- /dev/null +++ b/content/understanding-couchbase/views/data-nonjson.dita @@ -0,0 +1,25 @@ + + + + Non-JSON Data + Non-JSON data is data that can not be parsed as a JSON document and is stored as a binary copy. + + +

    Data that can not be parsed as JSON is always stored and returned as a binary copy of the + information submitted to the database. + For example, if an image is stored, the returned data is an identical binary copy of the stored image. + The + significance of the returned structure can be seen when editing the view via the Web Console.

    + +

    Non-JSON data is available as a base64 string during view processing. A non-JSON document can + be identified by examining the type field of the metadata structure + as well as during view processing by using the meta object supplied to the + map() function.

    + +

    Non-JSON data can be indexed through the views system by creating an index on the + key data. This is useful when the document key is significant. For example, + if you store information using a prefix to the key to identify the record type, you can + create document-type specific indexes.

    + +     +     diff --git a/content/understanding-couchbase/views/data-types.dita b/content/understanding-couchbase/views/data-types.dita new file mode 100644 index 0000000000..fb0870c3cd --- /dev/null +++ b/content/understanding-couchbase/views/data-types.dita @@ -0,0 +1,37 @@ + + + + Data Types + This section describes how the views system relates to stored data. + +

    The Couchbase query system relies on the information being formatted as a JSON document. The formatting + of the data in this form enables the individual fields of the data to be identified and + used.

    +

    When data is stored into the Couchbase database, the data is parsed. + If the data is identified as valid JSON, then the data is tagged and identified as valid JSON. + If the information cannot be parsed as valid JSON, then it is + stored as a verbatim binary copy of the submitted data.

    + + + + When retrieving the stored data, the format of the information depends on whether the data + was tagged as valid JSON or not. + + + + + + + + + + + + + + + + + +     diff --git a/content/understanding-couchbase/views/images/custom-reduce.png b/content/understanding-couchbase/views/images/custom-reduce.png new file mode 100644 index 0000000000..355d40b690 Binary files /dev/null and b/content/understanding-couchbase/views/images/custom-reduce.png differ diff --git a/content/understanding-couchbase/views/images/custom-rereduce.png b/content/understanding-couchbase/views/images/custom-rereduce.png new file mode 100644 index 0000000000..a1e973151f Binary files /dev/null and b/content/understanding-couchbase/views/images/custom-rereduce.png differ diff --git a/content/understanding-couchbase/views/images/stream-based-views.png b/content/understanding-couchbase/views/images/stream-based-views.png new file mode 100644 index 0000000000..e34ce4ce1c Binary files /dev/null and b/content/understanding-couchbase/views/images/stream-based-views.png differ diff --git a/content/understanding-couchbase/views/images/view-building.png b/content/understanding-couchbase/views/images/view-building.png new file mode 100644 index 0000000000..cddb438284 Binary files /dev/null and b/content/understanding-couchbase/views/images/view-building.png differ diff --git a/content/understanding-couchbase/views/images/view-types-datastore.png b/content/understanding-couchbase/views/images/view-types-datastore.png new file mode 100644 index 0000000000..417afdd54f Binary files /dev/null and b/content/understanding-couchbase/views/images/view-types-datastore.png differ diff --git a/content/understanding-couchbase/views/images/view-types-workflow.png b/content/understanding-couchbase/views/images/view-types-workflow.png new file mode 100644 index 0000000000..0323a02881 Binary files /dev/null and b/content/understanding-couchbase/views/images/view-types-workflow.png differ diff --git a/content/understanding-couchbase/views/images/views-basic-overview-missing.png b/content/understanding-couchbase/views/images/views-basic-overview-missing.png new file mode 100644 index 0000000000..1a6424071f Binary files /dev/null and b/content/understanding-couchbase/views/images/views-basic-overview-missing.png differ diff --git a/content/understanding-couchbase/views/images/views-basic-overview.png b/content/understanding-couchbase/views/images/views-basic-overview.png new file mode 100644 index 0000000000..e40786ffbf Binary files /dev/null and b/content/understanding-couchbase/views/images/views-basic-overview.png differ diff --git a/content/understanding-couchbase/views/images/views-grouping.png b/content/understanding-couchbase/views/images/views-grouping.png new file mode 100644 index 0000000000..586f61aa48 Binary files /dev/null and b/content/understanding-couchbase/views/images/views-grouping.png differ diff --git a/content/understanding-couchbase/views/images/views-query-flow.png b/content/understanding-couchbase/views/images/views-query-flow.png new file mode 100644 index 0000000000..66a6dcdc77 Binary files /dev/null and b/content/understanding-couchbase/views/images/views-query-flow.png differ diff --git a/content/understanding-couchbase/views/images/views-stale-sequence-stale.png b/content/understanding-couchbase/views/images/views-stale-sequence-stale.png new file mode 100644 index 0000000000..53b8314a6e Binary files /dev/null and b/content/understanding-couchbase/views/images/views-stale-sequence-stale.png differ diff --git a/content/understanding-couchbase/views/images/views-stale-sequence-updateafter.png b/content/understanding-couchbase/views/images/views-stale-sequence-updateafter.png new file mode 100644 index 0000000000..235f053dd4 Binary files /dev/null and b/content/understanding-couchbase/views/images/views-stale-sequence-updateafter.png differ diff --git a/content/understanding-couchbase/views/images/views-stale-sequence-updatebefore.png b/content/understanding-couchbase/views/images/views-stale-sequence-updatebefore.png new file mode 100644 index 0000000000..787c4c855b Binary files /dev/null and b/content/understanding-couchbase/views/images/views-stale-sequence-updatebefore.png differ diff --git a/content/understanding-couchbase/views/images/views_replica.png b/content/understanding-couchbase/views/images/views_replica.png new file mode 100644 index 0000000000..478447add8 Binary files /dev/null and b/content/understanding-couchbase/views/images/views_replica.png differ diff --git a/content/understanding-couchbase/views/spatial-views.dita b/content/understanding-couchbase/views/spatial-views.dita new file mode 100644 index 0000000000..a6813b1963 --- /dev/null +++ b/content/understanding-couchbase/views/spatial-views.dita @@ -0,0 +1,38 @@ + + + + Spatial Views + Couchbase Server offers spatial views to query geo-spatial information. + + +

    A spatial view contains geometry data that can be queried to return information based on + whether the recorded geometries exist within a given multidimensional range.

    +

    Spatial views support a variety of different use cases including multidimensional analysis, + geo-data, and a combination. When analyzing multidimensional data, spatial views can operate on + numeric data and map categories to numbers. Spatial views support the GeoJSON format and allow + bounding-box queries on complex geometries that are represented as points, line-strings, and + polygons. Geospatial views also index arrays supporting flat, multidimensional arrays of min-max + coordinates, and combination of arrays and GeoJSON objects

    +

    Below is a spatial view written in JavaScript that emits GeoJSON within an array combined with + document attributes. + function (doc, meta) { + if (doc.geo !== undefined and doc.updated !== undefined) { + var geojson = {type: "Point", coordinates: [doc.geo.lon, doc.geo.lat]}; + // The `doc.updated` might be an ISO 8601 date like "2015-10-07T08:57" + var date = Date.parse(doc.updated); + + //The GeoJSON object must be first element + emit([geojson, date], doc.name); + } +} +

    + + + + + + + +     + +     diff --git a/content/understanding-couchbase/views/spatial-views.ditamap b/content/understanding-couchbase/views/spatial-views.ditamap new file mode 100644 index 0000000000..32e51a3a2b --- /dev/null +++ b/content/understanding-couchbase/views/spatial-views.ditamap @@ -0,0 +1,21 @@ + + + + Spatial Views + + + + + + + + + + + + + + + + + diff --git a/content/understanding-couchbase/views/sv-ex1-create.dita b/content/understanding-couchbase/views/sv-ex1-create.dita new file mode 100644 index 0000000000..5697e01b31 --- /dev/null +++ b/content/understanding-couchbase/views/sv-ex1-create.dita @@ -0,0 +1,23 @@ + + + + Creating a Spatial View Function + A spatial view function is created to create a multidimensional view file for GeoJSON + polygons. + +
    +

    To query based on the location and the area, create a multidimensional view function with + three dimensions. Based on the GeoJSON polygon data information, the first two + dimensions are the location and the third dimension is the area.

    +

    In the following example, a development view is created with the design document name of + _design/areas and a spatial view named locAndArea.

    + + function(doc, meta) { + if (doc.geometry && doc.properties + && doc.properties.area && doc.properties.name) { + emit([doc.geometry, doc.properties.area], doc.properties.name); + } +} +
    +     +     diff --git a/content/understanding-couchbase/views/sv-ex1-query-all.dita b/content/understanding-couchbase/views/sv-ex1-query-all.dita new file mode 100644 index 0000000000..b7e8e42d3d --- /dev/null +++ b/content/understanding-couchbase/views/sv-ex1-query-all.dita @@ -0,0 +1,60 @@ + + + + Querying All Data + This query retrieves all data based on the GeoJSON polygons example. + +
    Description + +

    The results are either viewable in the web console or are returned when the spatial view + is queried without any additional parameters. In both cases, all documents are + retrieved.

    + + There isn't a specific ordering, expect it to be arbitrary. Once the view + file is updated, the order might change. +
    + +
    HTTP method and URI + GET /_design/[design-doc]/_spatial/[spatial-name] +
    + +
    Syntax + +

    Curl syntax:

    + + curl http://[localhost]:8092/[bucket-name]/_design/[design-doc]/_spatial/[spatial-name] +
    + +
    Example +

    The following example is based on the GeoJSON polygon data and the associated spatial view function.

    + +

    Curl example:

    + curl http://10.5.2.54:8092/default/_design/areas/_spatial/locAndArea +
    + +
    Response + +

    Returns all of the data:

    + + {"total_rows":0,"rows":[ + {"id": "bermuda_triangle", + "key": [[-80.19,-64.73],[18.43,32.31],[1150180,1150180]], + "value": "Bermuda Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-64.73,32.31],[-80.19,25.76],[-66.09,18.43],[-64.73,32.31]]]}}, + {"id": "research_triangle", + "key": [[-79.04000000000001,-78.67],[35.78,36],[252,252]], + "value": "Research Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-78.93000000000001,36],[-78.67,35.78],[-79.04000000000001,35.9],[-78.93000000000001,36]]]}}, + {"id": "flemish_diamond", + "key":[[3.55,4.84],[50.73,51.3],[2947,2947]], + "value":"Flemish Diamond", + "geometry": {"type":"Polygon","coordinates":[[[3.55,51.08],[4.36,50.73],[4.84,50.85],[4.45,51.3],[3.55,51.08]]]}}}]} + +

    The individual rows contain the total_rows (which is always 0), document ID, the key that + got stored, the emitted value, and geometry. The key gets automatically calculated and is + the enclosing bounding box of the emitted geometry. When refining the query, the key is + always used for comparison.

    + +
    +     +     diff --git a/content/understanding-couchbase/views/sv-ex1-query-area.dita b/content/understanding-couchbase/views/sv-ex1-query-area.dita new file mode 100644 index 0000000000..9d9c44b57e --- /dev/null +++ b/content/understanding-couchbase/views/sv-ex1-query-area.dita @@ -0,0 +1,54 @@ + + + + Querying on the Area + This query retrieves data based on the GeoJSON polygon example and associated with a large square footage. + + +
    Description +

    In this example, all documents queried are associated with any really large area. For + example, the criteria could be any areas bigger than 10,000 square kilometers without + caring about a specific location or query for areas bigger than 10,000 square + kilometers without caring where they are.

    + +

    In this case, the existing view can be queried with wildcards on the location + (the first two dimensions) and an open range for the area.

    + +
    + + +
    HTTP method and URI + GET [bucket-name]/_design/[design-doc]/_spatial/[spatial-name] +
    + + +
    Syntax +

    Curl syntax:

    + curl http://[localhost]:8092/[bucket-name]/_design/[design-doc]/_spatial/[spatial-name]?start_range=[]&end_range=[] + +
    +
    Example +

    The following example is based on the GeoJSON polygon data and the associated spatial view function.

    +

    Curl example:

    + + curl http://localhost:8092/default/_design/areas/_spatial/locAndArea?start_range=[null,null,10000]&end_range=[null,null,null] +

    Alternatively, the query could have used + start_range=[-180,-90,10000]&end_range=[180,90,null] because + the longitudes and latitudes have those bounds.

    + +
    +
    Response +

    The results contain only the Bermuda Triangle:

    + + {"total_rows":0,"rows":[ + {"id": "bermuda_triangle", + "key": [[-80.19,-64.73],[18.43,32.31],[1150180,1150180]], + "value": "Bermuda Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-64.73,32.31],[-80.19,25.76],[-66.09,18.43],[-64.73,32.31]]]}}]} +
    + + + + +     +     diff --git a/content/understanding-couchbase/views/sv-ex1-query-east.dita b/content/understanding-couchbase/views/sv-ex1-query-east.dita new file mode 100644 index 0000000000..dd02442314 --- /dev/null +++ b/content/understanding-couchbase/views/sv-ex1-query-east.dita @@ -0,0 +1,57 @@ + + + + Querying on the East + This query retrieves data based on the GeoJSON polygons example and associated with the + eastern hemisphere. + +
    Description + +

    In this example, all documents that are on the eastern hemisphere are queried. + In addition, since the area is also emitted as third dimension, queries need to contain that as well. + As we don't care what the area is, we'll query it with the wildcard, null. So the final query is:

    + +

    This means the following coordinates are specified:

    +
      +
    • longitude (first dimension) greater than 0 and smaller than 180
      • +
    • latitude (second dimension) between -90 and 90.
      • +
    • null (third dimension) as a wildcard since, in this case, the third dimension doesn't matter.
      • + +
    + +

    The longitude and latitude are represented by the query parameters are start_range=[0,-90]&end_range=[180,90]. + If just those two query parameters are specified, an error message displays indicating that the dimensionality doesn't match. + That's why the null wildcard is used to represent the third dimension.

    + +
    + + +
    HTTP method and URI + GET [bucket-name]/_design/[design-doc]/_spatial/[spatial-name] +
    + +
    Syntax +

    Curl syntax:

    + curl http://[localhost]:8092/[bucket-name]/_design/[design-doc]/_spatial/[spatial-name]?start_range=[]&end_range=[] +
    + +
    Example +

    The following example is based on the GeoJSON polygon data and the associated spatial view function.

    +

    Curl example:

    + curl http://10.5.2.54:8092/default/_design/areas/_spatial/locAndArea?start_range=[0,-90,null]&end_range=[180,90,null] +
    + + +
    Response + +

    The results contain only the Flemish Diamond:

    + + {"total_rows":0,"rows":[ + {"id": "flemish_diamond", + "key":[[3.55,4.84],[50.73,51.3],[2947,2947]], + "value":"Flemish Diamond", + "geometry":{"type":"Polygon","coordinates":[[[3.55,51.08],[4.36,50.73],[4.84,50.85],[4.45,51.3],[3.55,51.08]]]}}]} + +
    +     +     diff --git a/content/understanding-couchbase/views/sv-ex1-query-nonintersect.dita b/content/understanding-couchbase/views/sv-ex1-query-nonintersect.dita new file mode 100644 index 0000000000..1402080727 --- /dev/null +++ b/content/understanding-couchbase/views/sv-ex1-query-nonintersect.dita @@ -0,0 +1,52 @@ + + + + Querying on Non-intersect + This query retrieves data based on the GeoJSON polygon example and on bounding box parameters that are within the query range. + +
    Description +

    Queries only filter on a bounding box level. This means that even if the actual geometry + doesn't intersect the query ranges but its bounding box does, it is still included in the result.

    +
    + + +
    HTTP method and URI + GET [bucket-name]/_design/[design-doc]/_spatial/[spatial-name] +
    + + +
    Syntax +

    Curl syntax:

    + curl http://[localhost]:8092/[bucket-name]/_design/[design-doc]/_spatial/[spatial-name]?start_range=[]&end_range=[] + +
    +
    Example +

    The following example is based on the GeoJSON polygon data and the associated spatial view function.

    + +

    Curl example:

    + curl http://10.5.2.54:8092/default/_design/areas/_spatial/locAndArea?start_range=[-84,29,null]&end_range=[-72,37,null] + +
    + + +
    Response + +

    The response shows that the ranges don't intersect the polygon of the Bermuda Triangle, but its enclosing bounding + box does which can be found in its key.

    + + {"total_rows":0,"rows":[ + {"id": "bermuda_triangle", + "key": [[-80.19,-64.73],[18.43,32.31],[1150180,1150180]], + "value": "Bermuda Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-64.73,32.31],[-80.19,25.76],[-66.09,18.43],[-64.73,32.31]]]}}, + {"id": "research_triangle", + "key": [[-79.04000000000001,-78.67],[35.78,36],[252,252]], + "value": "Research Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-78.93000000000001,36],[-78.67,35.78],[-79.04000000000001,35.9],[-78.93000000000001,36]]]}}]} + + + +
    + +     +     diff --git a/content/understanding-couchbase/views/sv-example1.dita b/content/understanding-couchbase/views/sv-example1.dita new file mode 100644 index 0000000000..496b70168e --- /dev/null +++ b/content/understanding-couchbase/views/sv-example1.dita @@ -0,0 +1,240 @@ + + + + Playing with GeoJSON Polygons + This example shows GeoJSON using the polygon type. + + + +
    Description +

    The GeoJSON input documents are GeoJSON containing geographic regions of different kind. + The Bermuda Triangle, the Flemish Diamond and the Research Triangle in North + Carolina.

    { + "type": "Feature", + "properties": { + "name": "Bermuda Triangle", + "area": 1150180 + }, + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [-64.73, 32.31], + [-80.19, 25.76], + [-66.09, 18.43], + [-64.73, 32.31] + ] + ] + } +} + + +{ + "type": "Feature", + "properties": { + "name": "Flemish Diamond", + "area": 2947 + }, + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [3.55, 51.08], + [4.36, 50.73], + [4.84, 50.85], + [4.45, 51.30], + [3.55, 51.08] + ] + ] + } +} + + +{ + "type": "Feature", + "properties": { + "name": "Research Triangle", + "area": 252 + }, + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [-78.93, 36.00], + [-78.67, 35.78], + [-79.04, 35.90], + [-78.93, 36.00] + ] + ] + } +} +
    +
    Creating a Spatial View Function +

    A spatial view function is created to create a multidimensional view file for GeoJSON + polygons.

    +

    To query based on the location and the area, create a multidimensional view function with + three dimensions. Based on the GeoJSON polygon data information, the first two + dimensions are the location and the third dimension is the area.

    +

    In the following example, a development view is created with the design document name of + _design/areas and a spatial view named locAndArea.

    + + function(doc, meta) { + if (doc.geometry && doc.properties + && doc.properties.area && doc.properties.name) { + emit([doc.geometry, doc.properties.area], doc.properties.name); + } + } +
    +
    Querying All Data +

    This query retrieves all data based on the GeoJSON polygons example.

    +

    Description

    +

    The results are either viewable in the web console or are returned when the spatial view + is queried without any additional parameters. In both cases, all documents are + retrieved.

    + + There isn't a specific ordering, expect it to be arbitrary. Once the view + file is updated, the order might change. +

    HTTP Method and URI

    + GET /_design/[design-doc]/_spatial/[spatial-name] +

    Syntax

    +

    Curl syntax:

    + + curl http://[localhost]:8092/[bucket-name]/_design/[design-doc]/_spatial/[spatial-name] + +

    Example

    +

    The following example is based on the GeoJSON polygon data and the associated spatial view function.

    + +

    Curl example:

    + curl http://10.5.2.54:8092/default/_design/areas/_spatial/locAndArea + +

    Response

    + {"total_rows":0,"rows":[ + {"id": "bermuda_triangle", + "key": [[-80.19,-64.73],[18.43,32.31],[1150180,1150180]], + "value": "Bermuda Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-64.73,32.31],[-80.19,25.76],[-66.09,18.43],[-64.73,32.31]]]}}, + {"id": "research_triangle", + "key": [[-79.04000000000001,-78.67],[35.78,36],[252,252]], + "value": "Research Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-78.93000000000001,36],[-78.67,35.78],[-79.04000000000001,35.9],[-78.93000000000001,36]]]}}, + {"id": "flemish_diamond", + "key":[[3.55,4.84],[50.73,51.3],[2947,2947]], + "value":"Flemish Diamond", + "geometry": {"type":"Polygon","coordinates":[[[3.55,51.08],[4.36,50.73],[4.84,50.85],[4.45,51.3],[3.55,51.08]]]}}}]} + +

    The individual rows contain the total_rows (which is always 0), document ID, the key that + got stored, the emitted value, and geometry. The key gets automatically calculated and is + the enclosing bounding box of the emitted geometry. When refining the query, the key is + always used for comparison.

    +
    + +
    Querying on the Area +

    This query retrieves data based on the GeoJSON polygon example and associated with a large square footage.

    +

    Description

    +

    In this example, all documents queried are associated with any really large area. For + example, the criteria could be any areas bigger than 10,000 square kilometers without + caring about a specific location or query for areas bigger than 10,000 square + kilometers without caring where they are.

    + +

    In this case, the existing view can be queried with wildcards on the location + (the first two dimensions) and an open range for the area.

    +

    HTTP method and URI

    + GET [bucket-name]/_design/[design-doc]/_spatial/[spatial-name] + +

    Syntax

    +

    Curl syntax:

    + curl http://[localhost]:8092/[bucket-name]/_design/[design-doc]/_spatial/[spatial-name]?start_range=[]&end_range=[] + +

    Example

    +

    The following example is based on the GeoJSON polygon data and the associated spatial view function.

    +

    Curl example:

    + + curl http://localhost:8092/default/_design/areas/_spatial/locAndArea?start_range=[null,null,10000]&end_range=[null,null,null] +

    Alternatively, the query could have used + start_range=[-180,-90,10000]&end_range=[180,90,null] because + the longitudes and latitudes have those bounds.

    + +

    Response

    +

    The results contain only the Bermuda Triangle:

    + + {"total_rows":0,"rows":[ + {"id": "bermuda_triangle", + "key": [[-80.19,-64.73],[18.43,32.31],[1150180,1150180]], + "value": "Bermuda Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-64.73,32.31],[-80.19,25.76],[-66.09,18.43],[-64.73,32.31]]]}}]} +
    + +
    Querying on the East +

    This query retrieves data based on the GeoJSON polygons example and associated with the + eastern hemisphere.

    +

    Description

    +

    In this example, all documents that are on the eastern hemisphere are queried. + In addition, since the area is also emitted as third dimension, queries need to contain that as well. + As we don't care what the area is, we'll query it with the wildcard, null. So the final query is:

    + +

    This means the following coordinates are specified:

    +
      +
    • longitude (first dimension) greater than 0 and smaller than 180
      • +
    • latitude (second dimension) between -90 and 90.
      • +
    • null (third dimension) as a wildcard since, in this case, the third dimension doesn't matter.
      • + +
    + +

    The longitude and latitude are represented by the query parameters are start_range=[0,-90]&end_range=[180,90]. + If just those two query parameters are specified, an error message displays indicating that the dimensionality doesn't match. + That's why the null wildcard is used to represent the third dimension.

    + +

    HTTP method and URI

    + GET [bucket-name]/_design/[design-doc]/_spatial/[spatial-name] + +

    Syntax

    +

    Curl syntax:

    + curl http://[localhost]:8092/[bucket-name]/_design/[design-doc]/_spatial/[spatial-name]?start_range=[]&end_range=[] + +

    Example

    +

    The following example is based on the GeoJSON polygon data and the associated spatial view function.

    +

    Curl example:

    + curl http://10.5.2.54:8092/default/_design/areas/_spatial/locAndArea?start_range=[0,-90,null]&end_range=[180,90,null] + +

    Response

    +

    The results contain only the Flemish Diamond:

    + + {"total_rows":0,"rows":[ + {"id": "flemish_diamond", + "key":[[3.55,4.84],[50.73,51.3],[2947,2947]], + "value":"Flemish Diamond", + "geometry":{"type":"Polygon","coordinates":[[[3.55,51.08],[4.36,50.73],[4.84,50.85],[4.45,51.3],[3.55,51.08]]]}}]} +
    + +
    Querying on Non-intersect +

    This query retrieves data based on the GeoJSON polygon example and on bounding box parameters that are within the query range.

    +

    Description

    +

    Queries only filter on a bounding box level. This means that even if the actual geometry + doesn't intersect the query ranges but its bounding box does, it is still included in the result.

    +

    HTTP method and URI

    + GET [bucket-name]/_design/[design-doc]/_spatial/[spatial-name] +

    Syntax

    +

    Curl syntax:

    + curl http://[localhost]:8092/[bucket-name]/_design/[design-doc]/_spatial/[spatial-name]?start_range=[]&end_range=[] +

    Example

    +

    The following example is based on the GeoJSON polygon data and the associated spatial view function.

    + +

    Curl example:

    + curl http://10.5.2.54:8092/default/_design/areas/_spatial/locAndArea?start_range=[-84,29,null]&end_range=[-72,37,null] +

    Response

    +

    The response shows that the ranges don't intersect the polygon of the Bermuda Triangle, but its enclosing bounding + box does which can be found in its key.

    + + {"total_rows":0,"rows":[ + {"id": "bermuda_triangle", + "key": [[-80.19,-64.73],[18.43,32.31],[1150180,1150180]], + "value": "Bermuda Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-64.73,32.31],[-80.19,25.76],[-66.09,18.43],[-64.73,32.31]]]}}, + {"id": "research_triangle", + "key": [[-79.04000000000001,-78.67],[35.78,36],[252,252]], + "value": "Research Triangle", + "geometry": {"type":"Polygon","coordinates":[[[-78.93000000000001,36],[-78.67,35.78],[-79.04000000000001,35.9],[-78.93000000000001,36]]]}}]} +
    + +     +     diff --git a/content/understanding-couchbase/views/sv-example2.dita b/content/understanding-couchbase/views/sv-example2.dita new file mode 100644 index 0000000000..0fa21b420c --- /dev/null +++ b/content/understanding-couchbase/views/sv-example2.dita @@ -0,0 +1,122 @@ + + + + Playing with Non-geographic Spatial Views + This example shows a general spatial view implementation with non-geographic + data. + + +
    Description +

    Since spatial views use a general implementation of a multidimensional view file, they can + also be used for non-geographic data. This example uses a two-dimensional index, but + additional dimensions can be implemented.

    + +

    The scenario is querying log messages. In this example the log files have the format of the Apache 2.2 Error + Log. Each line of the log is a document and is stored as a single string.

    + + "[Mon Feb 02 12:53:35 2015] [info] Server built: Feb 1 2015 02:04:51" +"[Thu Feb 19 16:02:40 2015] [error] [client 199.193.192.229] File does not exist: /var/www/example.com/favicon.ico" +"[Thu Feb 19 20:33:23 2015] [debug] mod_deflate.c(615): [client 199.193.192.229] Zlib: Compressed 264 to 202 : URL /robots.txt" +"[Fri Feb 20 04:05:19 2015] [debug] mod_alias.c(482): [client 199.193.192.229] incomplete redirection target of '/cgi-bin/blog/index.cgi' for URI '/index.html' modified to 'http://example.com/cgi-bin/blog/index.cgi'" +"[Fri Feb 20 07:40:42 2015] [alert] No active workers found... Apache is exiting!" +"[Tue Feb 24 06:52:21 2015] [warn] NameVirtualHost *:80 has no VirtualHosts" +"[Tue Feb 24 18:12:05 2015] [error] [client 199.193.192.229] client denied by server configuration: /var/www/example.com" +
    + +
    Creating a spatial view + +

    A spatial view and view file is created containing the time and severity level. The spatial + view function could be:

    + +function(doc, meta) { + var severityToInt = { + emerg: 7, + alert: 6, + crit: 5, + error: 4, + warn: 3, + notic: 2, + info: 1, + debug: 0 + } + + var parts = doc.match(/\[.*?\]/g); + // Remove the brackets + parts = parts.map(function(part) {return part.slice(1, -1)}); + // Parse the date as timestamp in milliseconds since 1970-01-01 + var date = Date.parse(parts[0]); + // The emitted key needs to have numeric values only + var severity = severityToInt[parts[1]]; + // The part of the log message that isn't date or severity + var rest = doc.split(/(] )/).slice(4).join('') + + emit([date, severity], rest); +} + +
    + +
    Query: log messages from date + +

    Below are some sample queries on the data set. For brevity only the resulting values are + mentioned.

    + +

    Get all log messages from Feb 20 or newer:

    + + start_range=[1424386800000,null]&end_range=[null,null] + + "value":"NameVirtualHost *:80 has no VirtualHosts"}, +... +"value":"[client 199.193.192.229] client denied by server configuration: /var/www/example.com"}, +... +"value":"No active workers found... Apache is exiting!"}, +... +"value":"mod_alias.c(482): [client 199.193.192.229] incomplete redirection target of '/cgi-bin/blog/index.cgi' for URI '/index.html' modified to 'http://example.com/cgi-bin/blog/index.cgi'"} +... + +
    + +
    Query: log messages with severity +

    Get all log messages with severity error:

    + + start_range=[null,4]&end_range=[null,4] + + ... +"value":"[client 199.193.192.229] client denied by server configuration: /var/www/example.com"}, +... +"value":"[client 199.193.192.229] File does not exist: /var/www/example.com/favicon.ico"} +... + + +

    Get all log messages with severity warn or less (warn, + notic, info, debug):

    + + start_range=[null,0]&end_range=[null,3] + + .... +"value":"Server built: Feb 1 2015 02:04:51"}, +... +"value":"NameVirtualHost *:80 has no VirtualHosts"}, +... +"value":"mod_alias.c(482): [client 199.193.192.229] incomplete redirection target of '/cgi-bin/blog/index.cgi' for URI '/index.html' modified to 'http://example.com/cgi-bin/blog/index.cgi'"}, +... +"value":"mod_deflate.c(615): [client 199.193.192.229] Zlib: Compressed 264 to 202 : URL /robots.txt"} +... +
    + +
    Query: log messages from date and with severity +

    Get all log messages between Jan 1 and Feb 22 with severity between error and alert (error, crit, alert):

    + + start_range=[1420066800000,4]&end_range=[1424645999000,6] + + ... +"value":"No active workers found... Apache is exiting!"}, +... +"value":"[client 199.193.192.229] File does not exist: /var/www/example.com/favicon.ico"} +... + +
    + + +     +     diff --git a/content/understanding-couchbase/views/sv-queries-bbox.dita b/content/understanding-couchbase/views/sv-queries-bbox.dita new file mode 100644 index 0000000000..cd975be3e8 --- /dev/null +++ b/content/understanding-couchbase/views/sv-queries-bbox.dita @@ -0,0 +1,73 @@ + + + + Bounding Box Queries + Bounding box queries are implemented via HTTP method and URI. + + Use of the bounding box parameter is discouraged. Use the + start_range and end_range parameters instead. Every + bounding box can be expressed with start_range and + end_range parameters. + +

    If a bounding box is not supplied, the full data set is returned. When querying a spatial + index, use the bounding box to specify the boundaries of the query lookup on a given value. + The specification should be in the form of a comma-separated list of the coordinates to use + during the query.

    + +

    These coordinates are specified as in the GeoJSON specification, so the first + two numbers are the lower left coordinates, and the last two numbers are the upper + right coordinates.

    + +

    A bounding box can be expressed as with start_range and end_range parameters. Example:

    + bbox=0,0,180,90 + start_range=[0,0]&end_range[180,90] + +
    Syntax + + GET http://[localhost]:8092/places/_design/[design-doc]/_spatial/points?bbox=-180,-90,0,0 +Content-Type: application/json + +
    + + +
    Example +

    HTTP request example:

    + GET http://127.0.0.1:8092/places/_design/main/_spatial/points?bbox=-180,-90,0,0 +Content-Type: application/json +
    + + + +
    Response + +

    Example response:

    + + { + "total_rows": 0, + "rows": [ + { + "id": "oakland", + "key": [ + [ + -122.270833, + -122.270833 + ], + [ + 37.804444, + 37.804444 + ] + ], + "value": [ + "oakland", + [ + -122.270833, + 37.804444 + ] + ] + } + ] +} +
    + +     +     diff --git a/content/understanding-couchbase/views/sv-queries-closed-range.dita b/content/understanding-couchbase/views/sv-queries-closed-range.dita new file mode 100644 index 0000000000..08bf1ad92f --- /dev/null +++ b/content/understanding-couchbase/views/sv-queries-closed-range.dita @@ -0,0 +1,27 @@ + + + + Closed Range Queries + Closed range queries use the start_range and end_range parameters + with the bounds specified. + +

    Closed range queries are used to query items with a certain range. If no range is supplied, the + full data set is returned. For example, if only the longitude (1st dimension) and the + latitude (2nd dimension) is emitted, the bounds of a country could be queried.

    + +

    For example, to query shops in Germany that are open between 10:00 and 20:00.

    + +

    In this case, the emit could be:

    + emit([{ + "type": "Point", + "coordinates":[10.9, 48.4] +}, [1000, 2000]], null); + +

    This emit cannot be a query with a bounding box because it contains three + dimensions.

    + +

    The query for the shop emit could be:

    + ?start_range=[5.87,47.27,1000]&end_range=[15.04,55.06,2000] + +     +     diff --git a/content/understanding-couchbase/views/sv-queries-open-range.dita b/content/understanding-couchbase/views/sv-queries-open-range.dita new file mode 100644 index 0000000000..c1d0c97d9a --- /dev/null +++ b/content/understanding-couchbase/views/sv-queries-open-range.dita @@ -0,0 +1,27 @@ + + + + Open Range Queries + Open range queries specify null as a value on either one or both sides of the range. + +

    For example, to query shops in Germany that are open between 10:00 and 20:00.

    +

    In this case, the emit could be:

    + emit([{ + "type": "Point", + "coordinates":[10.9, 48.4] +}, [1000, 2000]], null); + +

    To query for shops in Germany with an opening time of 10:00 and no closing time:

    + ?start_range=[5.87,47.27,1000]&end_range=[15.04,55.06,null] + +

    To query for shops in Germany with no opening time and a closing time of 20:00:

    + ?start_range=[5.87,47.27,null]&end_range=[15.04,55.06,2000] + +

    To query for shops in Germany with no opening or closing time:

    + ?start_range=[5.87,47.27,null]&end_range=[15.04,55.06,null] + +

    To query for shops anywhere (no location specified) with an opening time of 10:00 and a closing time of 20:00:

    + ?start_range=[null,null,1000]&end_range=[null,null,2000] + +     +     diff --git a/content/understanding-couchbase/views/sv-query-parameters.dita b/content/understanding-couchbase/views/sv-query-parameters.dita new file mode 100644 index 0000000000..40be9714d6 --- /dev/null +++ b/content/understanding-couchbase/views/sv-query-parameters.dita @@ -0,0 +1,275 @@ + + + + Querying Spatial Views + Spatial views are retrieved with the REST API GET + /[bucket-name]/_design/[design-doc]/_spatial/[spatial-name] HTTP method and + URI. + + +
    HTTP method and URI + GET /[bucket-name]/_design/[design-doc]/_spatial/[spatial-name] + + + + + + + + HTTP + Description + + + + + Method and URI + GET + /[bucket-name]/_design/[design-doc]/_spatial/[spatial-name] + + + Request Data + None + + + Response Data + JSON of the documents returned by the view + + + Authentication Required + no + + + +
    + + + + + The start_range and end_range parameters + take a JSON array and are preferred over the bbox parameter. + + + Query parameters + + + + + + + Parameter + Type + Description + + + + + start_range + Array of numeric or null; optional + The number of elements must match the number of dimensions of the + index. + + + end_range + Array of numeric or null; optional + The number of elements must match the number of dimensions of the + index. + + + bbox + String; optional + Specify the bounding box for a spatial query. + + + limit + Numeric; optional + Limit the number of the returned documents to the specified number. + + + skip + Numeric; optional + Skip this number of records before starting to return the results. + + + stale + String; optional + Specifies the level of data freshness.

    Supported values:

      +
    • false

      The server waits for the indexer to finish the + changes that correspond to the current key-value document set and then + returns the latest entries from the view file.

      • +
    • ok

      The server returns the current entries from the view + file.

      • +
    • update_after

      The server returns the current entries from + the view file, and then initiates an view file update.

      • +

    +     +     + + +
    + +

    When submitting a view query, the stale parameter is used to specify the data freshness. + The stale parameter the following values:

    + +
      +
    • ok—The server returns the current entries from the index file.
      • +
    • update_after—The server returns the current entries from the index, + and then initiates an index update.
      • +
    • false—The server waits for the indexer to finish the changes that + correspond to the current key-value document set and then returns the latest entries + from the view index.
      • +
    + +

    Every 5 seconds the automatic update process checks whether 5000 changes have occurred. If a + minimum of 5000 changes occurred, an view file update is triggered. Otherwise, no update is + triggered. When triggered, the indexer requests from DCP all changes since it was last run. + The default number of changes to check for is 5000, but that number can be configured by + setting the updateMinChanges option. The update interval can also be + configured by setting the updateInterval option.

    + +

    The stale=false view query argument has been enhanced. When an application + sends a query that has the stale parameter set to false, the + application receives all recent changes to the documents, including changes that haven't + yet been persisted to disk. It considers all document changes that have been received at + the time the query was received.

    +

    You can issue the stale=false view query anytime and results will fetch + all changes that have been made when the query was issued.

    + + + For better scalability and throughput, set the + value of the stale parameter to ok. With the stream-based + views, data returned when stale is set to ok is closer to + the key-value data, even though it might not include all of it. + +
    + +
    Responses +

    The standard response includes total_rows, id, + key, value, and geometry. The + total_rows item is always zero. The geometry item + is not shown if no geometry was emitted. If a geometry was emitted, the + key contains the ranges of the calculated bounding box.

    + {"total_rows":0, "rows": [{ + "id":"id-name" + "key": [[value, value], [value, value]], + "value": null, + "geometry": {"type": "Point", "coordinates": [value, value]} +}]} + + +

    Example response if geometry was emitted:

    + {"total_rows":0, "rows": [{ + "id":"Augsburg" + "key": [[10.9, 10.9], [48.4, 48.4]], + "value": null, + "geometry": {"type": "Point", "coordinates": [10.9, 48.4]} +}]} + +

    Example response if geometry was not emitted:

    + {"total_rows":0, "rows": [{ + "id":"Augsburg" + "key": [[10.9, 10.9], [48.4, 48.4]], [1000, 2000]], + "value": null, +}]} + +
    +
    Open Range Queries +

    Open range queries specify null as a value on either one or both sides of the range.

    +

    For example, to query shops in Germany that are open between 10:00 and 20:00.

    +

    In this case, the emit could be:

    + emit([{ + "type": "Point", + "coordinates":[10.9, 48.4] + }, [1000, 2000]], null); + +

    To query for shops in Germany with an opening time of 10:00 and no closing time:

    + ?start_range=[5.87,47.27,1000]&end_range=[15.04,55.06,null] + +

    To query for shops in Germany with no opening time and a closing time of 20:00:

    + ?start_range=[5.87,47.27,null]&end_range=[15.04,55.06,2000] + +

    To query for shops in Germany with no opening or closing time:

    + ?start_range=[5.87,47.27,null]&end_range=[15.04,55.06,null] + +

    To query for shops anywhere (no location specified) with an opening time of 10:00 and a closing time of 20:00:

    + ?start_range=[null,null,1000]&end_range=[null,null,2000] +
    +
    Closed Range Queries +

    Closed range queries use the start_range and end_range parameters + with the bounds specified.

    +

    Closed range queries are used to query items with a certain range. If no range is supplied, the + full data set is returned. For example, if only the longitude (1st dimension) and the + latitude (2nd dimension) is emitted, the bounds of a country could be queried.

    + +

    For example, to query shops in Germany that are open between 10:00 and 20:00.

    + +

    In this case, the emit could be:

    + emit([{ + "type": "Point", + "coordinates":[10.9, 48.4] + }, [1000, 2000]], null); + +

    This emit cannot be a query with a bounding box because it contains three + dimensions.

    + +

    The query for the shop emit could be:

    + ?start_range=[5.87,47.27,1000]&end_range=[15.04,55.06,2000] +
    +
    Bounding Box Queries +

    Bounding box queries are implemented via HTTP method and URI.

    + Use of the bounding box parameter is discouraged. Use the + start_range and end_range parameters instead. Every + bounding box can be expressed with start_range and + end_range parameters. + +

    If a bounding box is not supplied, the full data set is returned. When querying a spatial + index, use the bounding box to specify the boundaries of the query lookup on a given value. + The specification should be in the form of a comma-separated list of the coordinates to use + during the query.

    + +

    These coordinates are specified as in the GeoJSON specification, so the first + two numbers are the lower left coordinates, and the last two numbers are the upper + right coordinates.

    + +

    A bounding box can be expressed as with start_range and end_range parameters. Example:

    + bbox=0,0,180,90 + start_range=[0,0]&end_range[180,90] +

    Syntax

    + GET http://[localhost]:8092/places/_design/[design-doc]/_spatial/points?bbox=-180,-90,0,0 + Content-Type: application/json + +

    Example

    +

    HTTP request example:

    + GET http://127.0.0.1:8092/places/_design/main/_spatial/points?bbox=-180,-90,0,0 + Content-Type: application/json +

    Response

    +

    Example response:

    + + { + "total_rows": 0, + "rows": [ + { + "id": "oakland", + "key": [ + [ + -122.270833, + -122.270833 + ], + [ + 37.804444, + 37.804444 + ] + ], + "value": [ + "oakland", + [ + -122.270833, + 37.804444 + ] + ] + } + ] + } +
    + +     +     diff --git a/content/understanding-couchbase/views/sv-writing-views-functions.dita b/content/understanding-couchbase/views/sv-writing-views-functions.dita new file mode 100644 index 0000000000..980941cbb0 --- /dev/null +++ b/content/understanding-couchbase/views/sv-writing-views-functions.dita @@ -0,0 +1,46 @@ + + + + Writing Spatial View Functions + Spatial views use a single spatial function that is similar to the map function used in + MapReduce views. + +

    A major difference between MapReduce views and spatial views is that spatial views don't have + a map and a reduce function, but only a single so-called "spatial function". That function + is very similar to the map function in MapReduce views.

    +

    Spatial views use a single spatial function that is similar to the map function used in + MapReduce views.

    +

    To create a spatial view, use the emit() function. The emit() function + parameters are a key and a value. The key must be a multidimensional bounding box. There + are several ways to define multidimensional bounding boxes. For more information about + keys, see . The value can + be any arbitrary valid JSON.

    +

    Here is an example spatial view function that emits a point with its height:

    + function(doc, meta) { + if (doc.loc && doc.height && doc.name) { + emit([doc.loc.lon, doc.loc.lat, doc.height], doc.name); + } +} +

    You can put a spatial view function into a design document by using the object name + spatial to indicate the nature of the view definition. For example, the + following design document includes the previous function as a view named + points.

    + { + "spatial" : { + "points" : "function(doc, meta) {if (doc.loc && doc.height && doc.name) { + emit([doc.loc.lon, doc.loc.lat, doc.height], doc.name);}}" + } +} +

    If your document already contains GeoJSON encoded data, you can directly emit that geometry:

    + function(doc, meta) { + if (doc.geometry && doc.name) { + emit([doc.geometry], doc.name); + } +} +

    To execute the spatial query, use the design document format that uses the embedded spatial + function. For example, a design document named main that resides within + the bucket places is executed with this URL: + http://[localhost]:8092/places/_design/main/_spatial/points.

    +     + +     diff --git a/content/understanding-couchbase/views/sv-writing-views-keys.dita b/content/understanding-couchbase/views/sv-writing-views-keys.dita new file mode 100644 index 0000000000..0de23187eb --- /dev/null +++ b/content/understanding-couchbase/views/sv-writing-views-keys.dita @@ -0,0 +1,78 @@ + + + + Keys in Spatial View Functions + The key must be a multidimensional bounding box. + +

    The keys in a spatial view function can defined in several ways:

    +
    + Single values +

    When a single value is used in the key, it expands to a collapsed range.

    +

    For example, the [1, 2] value is stored as [[1, 1], [2, + 2]].

    +
    +
    + Ranges +

    Use ranges for keys, for example, for shops with certain opening hours such as 10:00 to 20:00. + This example uses timestamps and converts the values of 10:00 - 20:00 to 1000 and 2000.

    + emit([[1000, 2000]], null); +
    +
    + GeoJSON geometry +

    You can use a GeoJSON geometry in a key. It must be the first element of the array. The + bounding box is automatically calculated and used as the range.

    + +

    Couchbase Server supports the following GeoJSON objects:

    +
      +
    • Point
      • +
    • MultiPoint
      • +
    • LineString
      • +
    • MultiLineString
      • +
    • Polygon
      • +
    • MultiPolygon
      • +
    • GeometryCollection
      • + +
    +
    +
    Examples +

    In this example, the key is stored internally as [[10.9, 10.9], [48.4, 48.4]].

    + emit([{ + "type": "Point", + "coordinates":[10.9, 48.4] +}], null); +

    In this example, the key is stored internally as [[10.9, 10.9], [48.4, 48.4]].

    + emit([{ + "type": "Point", + "coordinates":[10.9, 48.4] +}, [1000, 2000], 5], null); + For backward compatibility, you can also emit a GeoJSON geometry that is not wrapped in an array. +

    The value can be any arbitrary valid JSON. Here is an example spatial view function that emits a point with its height: +

    + function(doc, meta) { + if (doc.loc && doc.height && doc.name) { + emit([doc.loc.lon, doc.loc.lat, doc.height], doc.name); + } +} +

    You can put a spatial view function into a design document by using the object name spatial to indicate the nature of + the view definition. For example, the following design document includes the previous function as a view named points.

    + { + "spatial" : { + "points" : "function(doc, meta) {if (doc.loc && doc.height && doc.name) { + emit([doc.loc.lon, doc.loc.lat, doc.height], doc.name);}}" + } +} +

    If your document already contains GeoJSON encoded data, you can directly emit that geometry: +

    + function(doc, meta) { + if (doc.geometry && doc.name) { + emit([doc.geometry], doc.name); + } +} +

    To execute the spatial query, use the design document format that uses the embedded spatial function. + For example, a design document named main that resides within the bucket places is executed with this URL:

    + http://[localhost]:8092/places/_design/main/_spatial/points. +
    + +     +     diff --git a/content/understanding-couchbase/views/sv-writing-views.dita b/content/understanding-couchbase/views/sv-writing-views.dita new file mode 100644 index 0000000000..2b5370155a --- /dev/null +++ b/content/understanding-couchbase/views/sv-writing-views.dita @@ -0,0 +1,17 @@ + + + + Writing Spatial Views + Spatial views use a single spatial function that is similar to the map function used in MapReduce views. + +

    A major difference between MapReduce views and spatial views is that spatial views don't have + a map and a reduce function. Spatial views have only a single so-called spatial + function, which is very similar to the map function in MapReduce views.

    +

    To create a spatial view, use the emit() function. The + emit() function parameters are a key and value. The key must be a + multidimensional bounding box.

    + + + +     +     diff --git a/content/understanding-couchbase/views/views-basics.dita b/content/understanding-couchbase/views/views-basics.dita new file mode 100644 index 0000000000..5ce7acc806 --- /dev/null +++ b/content/understanding-couchbase/views/views-basics.dita @@ -0,0 +1,42 @@ + + + + View Concepts + Views allow you to extract specific fields and information from data and create an index. + +

    A view performs the following on the Couchbase unstructured (or semi-structured) data:

    +
      +
    • Extract specific fields and information from the data files.
      • +
    • Produce a view index of the selected information.
      • +
    + +

    By using JSON, the process of selecting individual fields for output is easier. The resulting + generated structure is a view on the stored data. The view that is created during this process + lets you iterate, select and query the information in your database from the raw data + objects.

    + +

    The view definition specifies the format and content of the information generated for each + document. Because the process relies on the JSON fields, if the document is not JSON or the + requested field in the view does not exist, the information is ignored. This enables the view + to be created, even if some documents have minor errors or lack the relevant fields + altogether.

    +

    One of the benefits of a document database is the ability to change the format of documents + stored in the database at any time, without requiring a wholesale change to applications or a + costly schema update before doing so.

    + +

    In the following example, the view takes the Name, City and Salary fields from the documents + and creates a array of this information for each document in the view. A view is created by + iterating over every single document within the Couchbase bucket and outputting the + specified information. The resulting view file is stored for future use and updated with + new data when the view is accessed. The process is incremental and therefore has a low + ongoing impact on performance. Creating a new view on an existing large data set may take a + long time to build but updates to the data are quick.

    + +

    The following diagram shows a brief overview of this process:

    + + + + + +     diff --git a/content/understanding-couchbase/views/views-development.dita b/content/understanding-couchbase/views/views-development.dita new file mode 100644 index 0000000000..60ebb5145f --- /dev/null +++ b/content/understanding-couchbase/views/views-development.dita @@ -0,0 +1,43 @@ + + + + Development Views + Views are created in a development environment in order to control the impact of views prior to deployment. + +

    Due to the nature of the Couchbase cluster and because of the size of the datasets that can be + stored across a cluster, the impact of view development needs to be controlled. Creating a view + implies the creation of the index which could slow down the performance of your server while the + index is being generated. However, views also need to be built and developed using the actively + stored information.

    +

    To support both the creation and testing of views, and the deployment of views in production, + Couchbase Server supports two different view types, Development views and Production views. The two + view types work identically, but have different purposes and restrictions placed upon their + operation.

    +

    Development views are designed to be used while you are still selecting and designing your view + definitions. While a view is in development mode, views operate with the following + attributes:

    +
      +
    • By default the development view works on only a subset of the stored information. You can, + however, force the generation of a development view information on the full dataset.
      • +
    • Development views use live data from the selected Couchbase bucket, enabling you to develop + and refine your view in real-time on your production data.
      • +
    • Development views are not automatically rebuilt, and during a rebalance operation, + development views are not updated, even when when consistent views are enabled, as this relies + on the automated update mechanism. Updating development views in this way would waste system + resources.
      • +
    • Development views are fully editable and modifiable during their lifetime. You can change and + update the view definition for a development view at any time. During development of the view, + you can view and edit stored document to help develop the view definition.
      • +
    • Development views are accessed from client libraries through a different URL than production + views, making it easy to determine the view type and information during development of your + application.
      • +
    +

    Within the Web Console, the execution of a view by default occurs only over a subset of the + full set of documents stored in the bucket. You can elect to run the View over the full set using + the Web Console.

    +

    Because of the selection process, the reduced set of documents may not be fully representative + of all the documents in the bucket. You should always check the view execution over the full + set.

    + +     diff --git a/content/understanding-couchbase/views/views-intro.dita b/content/understanding-couchbase/views/views-intro.dita new file mode 100644 index 0000000000..727b5df966 --- /dev/null +++ b/content/understanding-couchbase/views/views-intro.dita @@ -0,0 +1,34 @@ + + + + Views + Couchbase views enable indexing and querying of data. + +

    A view creates an index on the data according to the defined format and structure. The view + consists of specific fields and information extracted from the objects in Couchbase.

    +

    Views are eventually consistent compared to the underlying stored documents. Documents are + included in views when the document data is persisted to disk. Documents with expiry times are + removed from indexes when the expiration pager operates to remove the document from the + database.

    +

    Views are used for a number of reasons, including:

    +
      +
    • Indexing and querying data from stored objects
      • +
    • Producing lists of data on specific object types
      • +
    • Producing tables and lists of information based on your stored data
      • +
    • Extracting or filtering information from the database
      • +
    • Calculating, summarizing or reducing the information on a collection of stored + data
      • +
    +

    Multiple views can be created which provides multiple indexes and routes into the stored data. + By exposing specific fields from the stored information, views enable the following:

      +
    • Creating and querying stored data
      • +
    • Performing queries and selection on the data
      • +
    • Paginating through the view output
      • +

    + +

    The View Builder provides an interface for creating views within the web console. Views can be + accessed by using a Couchbase client library to retrieve matching records.

    + + +     diff --git a/content/understanding-couchbase/views/views-mapreduce-intro.dita b/content/understanding-couchbase/views/views-mapreduce-intro.dita new file mode 100644 index 0000000000..b654083a6d --- /dev/null +++ b/content/understanding-couchbase/views/views-mapreduce-intro.dita @@ -0,0 +1,9 @@ + + + + MapReduce Views + + +

    Concept definition.

    +     +     diff --git a/content/understanding-couchbase/views/views-operation.dita b/content/understanding-couchbase/views/views-operation.dita new file mode 100644 index 0000000000..cb57d61237 --- /dev/null +++ b/content/understanding-couchbase/views/views-operation.dita @@ -0,0 +1,398 @@ + + + + View Operations + Views create indexes on your information that enable search and select operations on the data. + +

    All views within Couchbase operate as follows:

    +
      +
    • Views are updated as the document data is updated in memory. There may be a delay between + the document being created or updated and the document being updated within the view + depending on the client-side query parameters.
      • +
    • Documents that are stored with an expiry are not automatically removed until the + background expiry process removes them from the database. This means that expired documents + may still exist within the index.
      • +
    • Views are scoped within a design document, with each design document part of a single + bucket. A view can only access the information within the corresponding bucket.
      • +
    • View names must be specified using one or more UTF-8 characters. You cannot have a blank + view name. View names cannot have leading or trailing white-space characters (space, tab, + newline, or carriage-return).
      • +
    • Document IDs that are not UTF-8 encodable are automatically filtered and not included in + any view. The filtered documents are logged so that they can be identified.
      • +
    • If you have a long view request, use POST instead of GET.
      • +
    • Views can only access documents defined within their corresponding bucket. You cannot + access or aggregate data from multiple buckets within a given view.
      • +
    • Views are created as part of a design document, and each design document exists within the + corresponding named bucket.
        +
      • Each design document can have 0-n views.
        • +
      • Each bucket can contain 0-n design documents.
        • +
      • +
    • All the views within a single design document are updated when the update to a single view + is triggered. For example, a design document with three views updates all three views + simultaneously when one view is updated.
      • +
    • Updates can be triggered in two ways:
        +
      • +

        At the point of access or query by using the stale parameter.

        +
        • +
      • +

        Automatically by Couchbase Server based on the number of updated documents, or the + period since the last update. Automatic updates can be controlled either globally, or + individually on each design document.

        +
        • +
      • +
    • +

      Views are updated incrementally. The first time the view is accessed, all the documents + within the bucket are processed through the map/reduce functions. Each new access to the + view only processes the documents that have been added, updated, or deleted, since the + last time the view index was updated.

      +
      • +
    +

    In practice, this means that views are entirely incremental in nature. Updates to views are + typically quick as they only update changed documents. Ensure that views are updated by using + either the built-in automatic update system, through client-side triggering, or explicit + updates within your application framework.

    +
      +
    • +

      The view update process is incremental by nature. The information is only appended to the + index stored on the disk making sure that the index gets updated efficiently. Compaction + (including auto-compaction) will optimize the index size on disk and optimize the index + structure. An optimized index is more efficient to be updated and queried.

      +
      • +
    • +

      The entire view is recreated if the view definition has changed. Because this would have + a detrimental effect on live data, only development views can be modified.

      +
      • +
    +

    Design documents organize views, and indexes are created according to the design document. If + you change a single view in a design document that contains multiple views, you will + invalidate all the views and stored indexes within the design document. You will then have to + rebuild all corresponding views defined in that design document. Rebuilding of the index will + increase the I/O across the cluster, in addition to the I/O required for any active production + views.

    +
      +
    • +

      You can choose to update the result set from a view before you query it or after you + query. Or you can choose to retrieve the existing result set from a view when you query + the view. In this case, the results are possibly out of date, or stale.

      +
      • +
    • +

      The views engine creates an index is for each design document; this index contains the + results for all the views within that design document.

      +
      • +
    • +

      The index information stored on disk consists of the combination of both the key and + value information defined within your view. The key and value data is stored in the index + so that the information can be returned as quickly as possible. Views can then include a + reduce function and return the reduced information by extracting that data from the + index.

      +
      • +
    +

    The value and key information from the defined map function are stored in the index. The + overall index size can be larger than the stored data if the emitted key/value information is + larger than the original source document data.

    +
    + How expiration impacts views +

    Be aware that Couchbase Server does lazy expiration, that is, expired items are flagged as + deleted rather than being immediately erased. Couchbase Server has a maintenance process, + called expiry pager that periodically looks through all information and erase expired + items. The maintenance process runs every 60 minutes, by default, unless configured to run + at a different interval. When an item is requested, Couchbase Server removes the item + flagged for deletion and provides a response that the item does not exist.

    +

    The result set from a view will contain any items stored on disk that meet the + requirements of your views function. Therefore, the information not removed from the disk + may appear as part of a result set when you query a view.

    +

    Using Couchbase views, you can also perform the reduce functions on data, which perform + calculations or other aggregations of data. For instance, if you want to count the instances + of a type of object, use a reduce function. Once again, if an item is on disk, it will be + included in any calculation performed by your reduce functions. Based on this behavior due + to disk persistence, here are guidelines on handling expiration with views:

    +
      +
    • +

      Detect expired documents in result sets: The items not removed as part of the + expiry pager maintenance process are part of a result set returned by querying the view.

      +
      • +
    • Perform aggregations and calculations on data: In some cases, you can perform the + reduce function to aggregate and calculate data in Couchbase Server. In + this case, Couchbase Server takes pre-calculated values stored for an index and derives a + final result. Any expired items still on disk will be part of the reduction, which might + not be an issue for your final result if the ratio of expired items is proportionately low + compared to other items. For instance, if you have ten expired scores still on disk for an + average performed over 1 million players, there may be only a minimal level of difference + in the final result. However, if you have ten expired scores on disk for an average + performed over 20 players, you would get very different result than the average you would + expect.

      Run the expiry pager process more frequently to ensure that items that have + expired are not included in calculations used in the reduce function. We recommend an + interval of 10 minutes for the expiry pager on each node of a cluster. Do note that this + interval will have some slight impact on node performance as it will be performing + cleanup more frequently on the node.

      • +
    + +

    For more information about setting intervals for the maintenance process, refer to the + Couchbase command-line tool and review the examples on exp_pager_stime.

    +
    +
    + How views function in a cluster +

    Distributing data. If you familiar working with Couchbase Server, you know that the + server distributes data across different nodes in a cluster. If you have four nodes in a + cluster, on average, each node will contain about 25% of active data. If you use views with + Couchbase Server, the indexing process runs on all four nodes, and the four nodes will + contain roughly 25% of the results from indexing on disk. This index is called a partial + index since it is based on a subset of data within a cluster (shown in the + illustration).

    +

    Replicating data and Indexes. Couchbase Server also provides data replication; this + means that the server will replicate data from one node to another node. In case the first + node fails the second node can still handle requests for the data. To handle possible node + failure, you can specify that Couchbase Server also replicate a partial index for replicated + data. By default, each node in a cluster will have a copy of each design document and view + functions. If you make any changes to a views function, Couchbase Server will replicate this + change to all nodes in the cluster. The sever will generate indexes from views within a + single design document and store the indexes in a single file on each node in the + cluster:

    + + + +

    Couchbase Server can optionally create replica indexes on nodes that contain replicated + data; this is to prepare your cluster for a failover scenario. The server does not replicate + index information from another node. Instead, each node creates an index for the replicated + data it stores. The server recreates indexes using the replicated data on a node for each + defined design document and view. By providing replica indexes, the server enables you to + perform queries even in the event of node failure. You can specify whether Couchbase Server + creates replica indexes or not when you create a data bucket.

    +

    Query Time within a Cluster

    +

    When you query a view and thereby trigger the indexing process, you send that request to a + single node in the cluster. This node then distributes the request to all other nodes in the + cluster. Depending on the parameter sent in your query, each node will either send the most + current partial index at that node, will update the partial index and send it, or send the + partial index and update it on disk. Couchbase Server will collect and collate these partial + indexes and send this aggregate result to a client.

    +

    To handle errors when you perform a query, you can configure how the cluster behaves when + errors occur.

    +

    Queries During Rebalance or Failover

    +

    You can query an index during cluster rebalance and node failover operations. If you + perform queries during rebalancing or node failure, Couchbase Server will ensure that you + receive the query results that you would expect from a node as if there were no rebalance or + node failure.

    +

    During node rebalancing, you will get the same results you would get as if the data were + active data on a node and as if data were not being moved from one node to another. In other + words, this feature ensures you get query results from a node during rebalancing that are + consistent with the query results you would have received from the node before rebalance + started. This functionality operates by default in Couchbase Server. However, you can + optionally choose to disable it.

    +

    If querying of indexes during cluster rebalancing and node failover is enabled, cluster + rebalancing will take more time. However, it is not recommended that you disable this + functionality in production without thorough testing or you might observe inconsistent query + results.

    +
    +
    View performance

    View performance includes the time taken to update + the view, the time required for the view update to be accessed, and the time for the updated + information to be returned, depend on different factors. Your file system cache, frequency + of updates, and the time between updating document data and accessing (or updating) a view + will all impact performance.

    Some key notes and points are provided below:
      +
    • Index queries are always accessed from disk; indexes are not kept in RAM by Couchbase + Server. However, frequently used indexes are likely to be stored in the filesystem cache + used for caching information on disk. Increasing your filesystem cache, and reducing the + RAM allocated to Couchbase Server from the total RAM available will increase the RAM + available for the OS.
      • +
    • The filesystem cache will play a role in the update of the index information process. + Recently updated documents are likely to be stored in the filesystem cache. Requesting a + view update immediately after an update operation will likely use information from the + filesystem cache. The eventual persistence nature implies a small delay between updating a + document, it being persisted, and then being updated within the index.
      • +
    Keeping some RAM reserved for your operating system to allocate filesystem cache, or + increasing the RAM allocated to filesystem cache, will help keep space available for index + file caching.
      +
    • View indexes are stored, accessed, and updated, entirely independently of the document + updating system. The index update and retrieval are not dependent on documents in memory + to build the index information. Separate systems also mean that the performance, when + retrieving and accessing the cluster, is not dependent on the document store.
      • +
    +
    Index updates and the stale parameter

    Indexes are + created by Couchbase Server based on the view definition, but updating of these indexes can + be controlled at the point of data querying, rather than each time data is inserted. Whether + the index is updated when queried can be controlled through the stale + parameter.

    Irrespective of the stale parameter, documents can only be + indexed by the system once the document is persisted to the disk. If the document is not + persisted to the disk, use of stale will not force this process. You can + use the observe operation to monitor when documents are persisted to the + disk or updated in the index.

    Views can also be updated automatically according to a + document change, or interval count.

    Three values for stale are + supported:

      +
    • stale=ok
      • +

    The index is not updated. If an index exists for the given view, then the information + in the current index is used as the basis for the query and the results are returned + accordingly.

    + +

    This setting results in the fastest response times to a given query since the + existing index is used without being updated. However, this might return incomplete + information if changes have been made to the database, and these documents would otherwise + be included in the given view.

      +
    • stale=false
      • +

    The index is updated before you execute the query, making sure that any documents + updated and persisted to disk are included in the view. The client will wait until the index + has been updated before the query has executed and, therefore, the response will be delayed + until the updated index is available.

    + +
      +
    • stale=update_after
      • +

    This is the default setting if no stale parameter is specified. The + existing index is used as the basis of the query, but the index is marked for updating once + the results have been returned to the client.

    + +

    The indexing engine is an asynchronous process; this means querying an index may + produce results you may not expect. For example, if you update a document, and then + immediately run a query on that document you may not get the new information in the emitted + view data. The document updates have not yet been committed to the disk, at which point the + updates are indexed.

    The deleted documents may appear in the index even after deletion + because the deleted document is still not removed from the index.

    + + + + + +

    When you have multiple clients accessing an index, the + index update process and results returned to clients depend on the parameters passed by each + client and the sequence that the clients interact with the server.

      +
    • +

      Situation 1

      +
        +
      1. Client 1 queries view with stale=false
        2. +
      2. Client 1 waits until server updates the index
        3. +
      3. Client 2 queries view with stale=false while re-indexing from + Client 1 still in progress
        4. +
      4. Client 2 will wait until existing index process triggered by Client 1 completes. + Client 2 gets updated index.
        5. +
      +
      • +
    • +

      Situation 2

      +
        +
      1. Client 1 queries view with stale=false
        2. +
      2. Client 1 waits until server updates the index
        3. +
      3. Client 2 queries view with stale=ok while re-indexing from Client 1 + in progress
        4. +
      4. Client 2 will get the existing index
        5. +
      +
      • +
    • +

      Situation 3

      +
        +
      1. Client 1 queries view with stale=false
        2. +
      2. Client 1 waits until server updates the index
        3. +
      3. Client 2 queries view with stale=update_after
        4. +
      4. If re-indexing from Client 1 not done, Client 2 gets the existing index. If + re-indexing from Client 1 is done, Client 2 gets this updated index and triggers + re-indexing.
        5. +
      +
      • +
    Index updates may be stacked if multiple clients request the view be updated before the + information is returned ( stale=false ). Multiple clients updating and + querying the index data can then get the updated document and version of the view each time. + For stale=update_after queries, there is no stacking, since all updates occur + after the query has been accessed.

    Sequential accesses

      +
    1. Client 1 queries view with stale=ok
      2. +
    2. Client 2 queries view with stale=false
      3. +
    3. View gets updated
      4. +
    4. Client 1 queries a second time view with stale=ok
      5. +
    5. Client 1 gets the updated view version
      6. +

    The above scenario can cause problems when paginating over a number of records as the + record sequence might change between individual queries.

    +
    + Automated index updates +

    In addition to a configurable update interval, you can also update all indexes + automatically in the background. You configure automated update through two parameters, the + update time interval in seconds and the number of document changes that occur before the + views engine updates an index. These two parameters are updateInterval and + updateMinChanges.

    +
      +
    • +

      updateInterval: the time interval in milliseconds, the default is 5000 + milliseconds. At every updateInterval the views engine checks if the + number of document mutations on disk is greater than updateMinChanges. + If true, it triggers the view update. The documents stored on disk potentially lag + documents that are in-memory for tens of seconds.

      +
      • +
    • +

      updateMinChanges: the number of document changes that occur before + re-indexing occurs, the default is 5000 changes.

      +
      • +
    +

    The auto-update process only operates on the full-set development and production indexes. + Auto-update does not operate on partial set development indexes.

    +

    Irrespective of the automated update process, documents can only be indexed by the system + once the document has been persisted to disk. If the document has not been persisted to + disk, the automated update process will not force the unwritten data to be written to disk. + You can use the observe operation to monitor when documents have been + persisted to disk or updated in the index.

    +

    The updates are applied as follows:

    +
      +
    • Active indexes, Production views
      • +
    +

    For all active, production views, indexes are automatically updated according to the update + interval updateInterval and the number of document changes + updateMinChanges.

    +

    If updateMinChanges is set to 0 (zero), then automatic updates are + disabled for main indexes.

    +
      +
    • Replica indexes
      • +
    +

    If replica indexes have been configured for a bucket, the index is automatically updated + according to the document changes ( replicaUpdateMinChanges ; default 5000) + settings.

    +

    If replicaUpdateMinChanges is set to 0 (zero), then automatic updates are + disabled for replica indexes.

    +

    The trigger level can be configured both globally and for individual design documents for + all indexes using the REST API.

    + The ddocs allow you to set + updateMinChanges or replicaUpdateMinChanges only via + options. The updateInterval can only be set for the whole cluster. + + +

    To obtain the current view update daemon settings, access a node within the cluster on the + administration port using the URL + http://nodename:8091/settings/viewUpdateDaemon :

    + GET http://Administrator:Password@nodename:8091/settings/viewUpdateDaemon +

    The request returns the JSON of the current update settings:

    + { + "updateInterval":5000, + "updateMinChanges":5000, + "replicaUpdateMinChanges":5000 +} +

    To update the settings, use POST with a data payload that includes the + updated values. For example, to update the time interval to 10 seconds, and document changes + to 7000 each:

    + POST http://nodename:8091/settings/viewUpdateDaemon +updateInterval=10000&updateMinChanges=7000 +

    If successful, the return value is the JSON of the updated configuration.

    +

    To configure the updateMinChanges or + replicaUpdateMinChanges values explicitly on individual design documents, + specify the parameters within the options section of the design document. + For example:

    + { + "_id": "_design/myddoc", + "views": { + "view1": { + "map": "function(doc, meta) { if (doc.value) { emit(doc.value, meta.id);} }" + } + }, + "options": { + "updateMinChanges": 1000, + "replicaUpdateMinChanges": 20000 + } +} +

    You can set this information when creating and updating design documents through the design + document REST API. To perform this operation using the curl tool:

    + > curl -X POST -v -d 'updateInterval=7000&updateMinChanges=7000' \ + 'http://Administrator:Password@192.168.0.72:8091/settings/viewUpdateDaemon' + +

    Partial-set development views are not automatically rebuilt. During rebalancing development + views are not updated, even when consistent views are enabled, as this relies on the + automated update mechanism. Updating development views in this way would waste system + resources.

    +
    + +     diff --git a/content/understanding-couchbase/views/views-production.dita b/content/understanding-couchbase/views/views-production.dita new file mode 100644 index 0000000000..0ea15e110c --- /dev/null +++ b/content/understanding-couchbase/views/views-production.dita @@ -0,0 +1,65 @@ + + + + Production Views + Production views are used in a deployed environment. + +

    Due to the nature of the Couchbase cluster and because of the size of the datasets that can be + stored across a cluster, the impact of view development needs to be controlled. Creating a view + implies the creation of the index which could slow down the performance of your server while the + index is being generated. However, views also need to be built and developed using the actively + stored information.

    +

    To support both the creation and testing of views, and the deployment of views in production, + Couchbase Server supports two different view types, Development views and Production views. The two + view types work identically, but have different purposes and restrictions placed upon their + operation.

    +
      +
    • Production views
      • +
    +

    Production views are optimized for production use. A production view has the following + attributes:

    +
      +
    • Production views always operate on the full dataset for their respective bucket.
      • +
    • Production views can either be created from the Web Console or through REST API. From the Web + Console, you first create development views and then publish them as production views. Through + REST API, you directly create the production views (and skip the initial development + views).
      • +
    • Production views cannot be modified through the UI. You can only access the information + exposed through a production view. To make changes to a production view, it must be copied to a + development view, edited, and re-published.
      • +
    +

    Views can be updated by the REST API, but updating a production design document immediately + invalidates all of the views defined within it.

    +
      +
    • Production views are accessed through a different URL to development views.
      • +
    +

    The support for the two different view types means that there is a typical work flow for view + development, as shown in the figure below:

    + + + + + +

    The above diagram features the following steps:

    +
      +
    1. Create a development view and view the sample view output.
      2. +
    2. Refine and update your view definition to suit your needs, repeating the process until your + view is complete. During this phase you can access your view from your client library and + application to ensure it suits your needs.
      3. +
    3. Once the view definition is complete, apply your view to your entire Cluster dataset.
      4. +
    4. Push your development view into production. This moves the view from development into + production, and renames the index (so that the index does not need to be rebuilt).
      5. +
    5. Start using your production view.
      6. +
    +

    Individual views are created as part of a design document. Each design document can have + multiple views, and each Couchbase bucket can have multiple design documents. You can therefore + have both development and production views within the same bucket while you development different + indexes on your data.

    + +

    For information on publishing a view from development to production state.

    + + + + +     diff --git a/content/understanding-couchbase/views/views-production.ditamap b/content/understanding-couchbase/views/views-production.ditamap new file mode 100644 index 0000000000..6ab00d4006 --- /dev/null +++ b/content/understanding-couchbase/views/views-production.ditamap @@ -0,0 +1,36 @@ + + + + Views + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/understanding-couchbase/views/views-query-samples.dita b/content/understanding-couchbase/views/views-query-samples.dita new file mode 100644 index 0000000000..04ea89d055 --- /dev/null +++ b/content/understanding-couchbase/views/views-query-samples.dita @@ -0,0 +1,655 @@ + + + + View and Query Examples + This section provides general information and query examples. + +

    Building views and querying the indexes they generate is a combined process based both on the + document structure and the view definition. + Writing an effective view to query your data may + require changing or altering your document structure, or creating a more complex view in order + to allow the specific selection of the data through the querying mechanism.

    +

    For background and examples, the following selections provide a number of different scenarios + and examples have been built to demonstrate the document structures, views and querying + parameters required for different situations.

    + + + + +
    General advice

    There are some general points and advice for writing + all views that apply irrespective of the document structure, query format, or view + content.

      +
    • Do not assume the field exists in all documents.
      • +

    Fields may be missing from your document, or may only be supported in specific + document types. Use an if test to identify problems. For example:

    + + if (document.firstname)... +
      +
    • View output is case sensitive.
      • +

    The value emitted by the emit() function is case sensitive. Emitting + a field value of ‘Martin’ but specifying a key value of ‘martin’ will not + match the data. Emitted data, and the key selection values, should be normalized to + eliminate potential problems. For example:

    + emit(doc.firstname.toLowerCase(),null); +
      +
    • Number formatting
      • +

    Numbers within JavaScript may inadvertently be converted and output as strings. To + ensure that data is correctly formatted, the value should be explicitly converted. For + example:

    + emit(parseInt(doc.value,10),null);

    The + parseInt() built-in function will convert a supplied value to an integer. + The parseFloat() function can be used for floating-point + numbers.

    + + + +
    Validating document type +

    If your data set includes documents that may be either JSON or binary, then you do not want + to create a view that outputs individual fields for non-JSON documents. You can fix this + by using a view that checks the metadata type field before outputting + the JSON view information:

    function(doc,meta) { + if (meta.type == "json") { + emit(doc.firstname.toLowerCase(),null); + } +}

    In + the above example, the emit() function will only be called on a valid JSON + document. Non-JSON documents will be ignored and not included in the view + output.

    + + + +
    Document ID (primary) index +

    To create a ‘primary key’ index, i.e. an index that contains a list of every document + within the database, with the document ID as the key, you can create a simple + view:

    function(doc,meta) +{ + emit(meta.id,null); +} +

    This + enables you to iterate over the documents stored in the database.

    This will provide + you with a view that outputs the document ID of every document in the bucket using the + document ID as the key.

    The view can be useful for obtaining groups or ranges of + documents based on the document ID, for example to get documents with a specific ID + prefix:

    ?startkey="object"&endkey="object\u0000" +

    Or + to obtain a list of objects within a given + range:

    ?startkey="object100"&endkey="object199" +

    For + all views, the document ID is automatically included as part of the view response. But the + without including the document ID within the key emitted by the view, it cannot be used as a + search or querying mechanism.

    + + + +
    Secondary index +

    The simplest form of view is to create an index against a single field from the documents + stored in your database.

    For example, given the document + structure:

    { + "firstname": "Martin", + "lastname": "Brown" +} +

    A + view to support queries on the firstname field could be defined as + follows:

    function(doc, meta) +{ + if (doc.firstname) + { + emit(doc.firstname.toLowerCase(),null); + } +} +

    The + view works as follows for each document:

      +

    • Only outputs a record if the document contains a firstname + field.

      • +

    • Converts the content of the firstname field to lowercase.

      • +

    Queries can now be specified by supplying a string converted to lowercase. For + example:

    ?key="martin" +

    Will return all + documents where the firstname field contains ‘Martin’, regardless of the + document field capitalization.

    + + + +
    Using expiration metadata +

    The metadata object makes it very easy to create and update different views on your data + using information outside of the main document data. For example, you can use the expiration + field within a view to get the list of recently active sessions in a system.

    Using the + following map() function, which uses the expiration as part of the emitted + data.

    function(doc, meta) +{ + if (doc.type && doc.type == "session") + { + emit(meta.expiration, doc.nickname) + } +} +

    If + you have sessions which are saved with a TTL, this will allow you to give a view of who was + recently active on the service.

    + + + +
    Emitting multiple rows

    The emit() function is used to + create a record of information for the view during the map phase, but it can be called + multiple times within that map phase to allowing querying over more than one source of + information from each stored document.

    An example of this is when the source documents + contain an array of information. For example, within a recipe document, the list of + ingredients is exposed as an array of objects. By iterating over the ingredients, an index + of ingredients can be created and then used to find recipes by + ingredient.

    { + "title": "Fried chilli potatoes", + "preptime": "5" + "servings": "4", + "totaltime": "10", + "subtitle": "A new way with chips.", + "cooktime": "5", + "ingredients": [ + { + "ingredtext": "chilli powder", + "ingredient": "chilli powder", + "meastext": "3-6 tsp" + }, + { + "ingredtext": "potatoes, peeled and cut into wedges", + "ingredient": "potatoes", + "meastext": "900 g" + }, + { + "ingredtext": "vegetable oil for deep frying", + "ingredient": "vegetable oil for deep frying", + "meastext": "" + } + ], +} +

    The + view can be created using the following map() + function:

    function(doc, meta) +{ + if (doc.ingredients) + { + for (i=0; i < doc.ingredients.length; i++) + { + emit(doc.ingredients[i].ingredient, null); + } + } +} +

    To + query for a specific ingredient, specify the ingredient as a + key:

    ?key="carrot" +

    The keys + parameter can also be used in this situation to look for recipes that contain multiple + ingredients. For example, to look for recipes that contain either "potatoes" or "chilli + powder" you would + use:

    ?keys=["potatoes","chilli powder"] +

    This + will produce a list of any document containing either ingredient. A simple count of the + document IDs by the client can determine which recipes contain all three.

    The output + can also be combined. For example, to look for recipes that contain carrots and can be + cooked in less than 20 minutes, the view can be rewritten + as:

    function(doc, meta) +{ + if (doc.ingredients) + { + for (i=0; i < doc.ingredients.length; i++) + { + if (doc.ingredients[i].ingredtext && doc.totaltime) + { + emit([doc.ingredients[i].ingredtext, parseInt(doc.totaltime,10)], null); + } + } + } +} +

    In + this map function, an array is output that generates both the ingredient name, and the total + cooking time for the recipe. To perform the original query, carrot recipes requiring less + than 20 minutes to + cook:

    ?startkey=["carrot",0]&endkey=["carrot",20] +

    This + generates the following + view:

    {"total_rows":26471,"rows":[ +{"id":"Mangoandcarrotsmoothie","key":["carrots",5],"value":null}, +{"id":"Cheeseandapplecoleslaw","key":["carrots",15],"value":null} +] +} +
    + + + +
    Date and time selection

    For date and time selection, consideration + must be given to how the data will need to be selected when retrieving the information. This + is particularly true when you want to perform log roll-up or statistical collection by using + a reduce function to count or quantify instances of a particular event over + time.

    Examples of this in action include querying data over a specific range, on + specific day or date combinations, or specific time periods. Within a traditional relational + database it is possible to perform an extraction of a specific date or date range by storing + the information in the table as a date type.

    Within a map/reduce, the effect can be + simulated by exposing the date into the individual components at the level of detail that + you require. For example, to obtain a report that counts individual log types over a period + identifiable to individual days, you can use the following map() + function:

    function(doc, meta) { + emit([doc.year, doc.mon, doc.day, doc.logtype], null); +}

    By + incorporating the full date into the key, the view provides the ability to search for + specific dates and specific ranges. By modifying the view content you can simplify this + process further. For example, if only searches by year/month are required for a specific + application, the day can be omitted.

    And with the corresponding + reduce() built-in of _count, you can perform a number of + different queries. Without any form of data selection, for example, you can use the + group_level parameter to summarize down as far as individual day, month, + and year. Additionally, because the date is explicitly output, information can be selected + over a specific range, such as a specific + month:

    endkey=[2010,9,30]&group_level=4&startkey=[2010,9,0]

    Here + the explicit date has been specified as the start and end key. The + group_level is required to specify roll-up by the date and log + type.

    This will generate information similar to + this:

    {"rows":[ +{"key":[2010,9,1,"error"],"value":5}, +{"key":[2010,9,1,"warning"],"value":10}, +{"key":[2010,9,2,"error"],"value":8}, +{"key":[2010,9,2,"warning"],"value":9}, +{"key":[2010,9,3,"error"],"value":16}, +{"key":[2010,9,3,"warning"],"value":8}, +{"key":[2010,9,4,"error"],"value":15}, +{"key":[2010,9,4,"warning"],"value":11}, +{"key":[2010,9,5,"error"],"value":6}, +{"key":[2010,9,5,"warning"],"value":12} +] +}

    Additional + granularity, for example down to minutes or seconds, can be achieved by adding those as + further arguments to the map + function:

    function(doc, meta) +{ + emit([doc.year, doc.mon, doc.day, doc.hour, doc.min, doc.logtype], null); +}

    The + same trick can also be used to output based on other criteria. For example, by day of the + week, week number of the year or even by + period:

    function(doc, meta) { + if (doc.mon) + { + var quarter = parseInt((doc.mon - 1)/3,10)+1; + + emit([doc.year, quarter, doc.logtype], null); + } +}

    To + get more complex information, for example a count of individual log types for a given date, + you can combine the map() and reduce() stages to provide + the collation.

    For example, by using the following map() function we + can output and collate by day, month, or year as before, and with data selection at the date + level.

    function(doc, meta) { + emit([doc.year, doc.mon, doc.day], doc.logtype); +} +

    For + convenience, you may wish to use the dateToArray() function, which converts + a date object or string into an array. For example, if the date has been stored within the + document as a single + field:

    function(doc, meta) { + emit(dateToArray(doc.date), doc.logtype); +} +

    For + more information, see dateToArray().

    Using the following + reduce() function, data can be collated for each individual logtype for + each day within a single record of + output.

    function(key, values, rereduce) +{ + var response = {"warning" : 0, "error": 0, "fatal" : 0 }; + for(i=0; i<values.length; i++) + { + if (rereduce) + { + response.warning = response.warning + values[i].warning; + response.error = response.error + values[i].error; + response.fatal = response.fatal + values[i].fatal; + } + else + { + if (values[i] == "warning") + { + response.warning++; + } + if (values[i] == "error" ) + { + response.error++; + } + if (values[i] == "fatal" ) + { + response.fatal++; + } + } + } + return response; +}

    When + queried using a group_level of two (by month), the following output is + produced:

    {"rows":[ +{"key":[2010,7], "value":{"warning":4,"error":2,"fatal":0}}, +{"key":[2010,8], "value":{"warning":4,"error":3,"fatal":0}}, +{"key":[2010,9], "value":{"warning":4,"error":6,"fatal":0}}, +{"key":[2010,10],"value":{"warning":7,"error":6,"fatal":0}}, +{"key":[2010,11],"value":{"warning":5,"error":8,"fatal":0}}, +{"key":[2010,12],"value":{"warning":2,"error":2,"fatal":0}}, +{"key":[2011,1], "value":{"warning":5,"error":1,"fatal":0}}, +{"key":[2011,2], "value":{"warning":3,"error":5,"fatal":0}}, +{"key":[2011,3], "value":{"warning":4,"error":4,"fatal":0}}, +{"key":[2011,4], "value":{"warning":3,"error":6,"fatal":0}} +] +}

    The + input includes a count for each of the error types for each month. Note that because the key + output includes the year, month and date, the view also supports explicit querying while + still supporting grouping and roll-up across the specified group. For example, to show + information from 15th November 2010 to 30th April 2011 using the following + query:

    ?endkey=[2011,4,30]&group_level=2&startkey=[2010,11,15] +

    Which + generates the following + output:

    {"rows":[ +{"key":[2010,11],"value":{"warning":1,"error":8,"fatal":0}}, +{"key":[2010,12],"value":{"warning":3,"error":4,"fatal":0}}, +{"key":[2011,1],"value":{"warning":8,"error":2,"fatal":0}}, +{"key":[2011,2],"value":{"warning":4,"error":7,"fatal":0}}, +{"key":[2011,3],"value":{"warning":4,"error":4,"fatal":0}}, +{"key":[2011,4],"value":{"warning":5,"error":7,"fatal":0}} +] +}

    Keep + in mind that you can create multiple views to provide different views and queries on your + document data. In the above example, you could create individual views for the limited + datatypes of logtype to create a warningsbydate view.

    + + + +
    Selective record output

    If you are storing different document types + within the same bucket, then you may want to ensure that you generate views only on a + specific record type within the map() phase. This can be achieved by using + an if statement to select the record.

    For example, if you are storing + blog ‘posts’ and ‘comments’ within the same bucket, then a view on the blog posts could be + created using the following + map:

    function(doc, meta) { + if (doc.title && doc.type && doc.date && + doc.author && doc.type == 'post') + { + emit(doc.title, [doc.date, doc.author]); + } +} +

    The + same solution can also be used if you want to create a view over a specific range or value + of documents while still allowing specific querying structures. For example, to filter all + the records from the statistics logging system over a date range that are of the type error + you could use the following map() + function:

    function(doc, meta) { + if (doc.logtype == 'error') + { + emit([doc.year, doc.mon, doc.day],null); + } +} +

    The + same solution can also be used for specific complex query types. For example, all the + recipes that can be cooked in under 30 minutes, made with a specific + ingredient:

    function(doc, meta) +{ + if (doc.totaltime &amp;&amp; doc.totaltime <= 20) + { + if (doc.ingredients) { + for (i=0; i < doc.ingredients.length; i++) + { + if (doc.ingredients[i].ingredtext) + { + emit(doc.ingredients[i].ingredtext, null); + } + } + } + } +} +

    The + above function provides for much quicker and simpler selection of recipes by using a query and + the key parameter, instead of having to work out the range that may be + required to select recipes when the cooking time and ingredients are generated by the + view.

    These selections are application specific, but by producing different views for + a range of appropriate values, for example 30, 60, or 90 minutes, recipe selection can be + much easier at the expense of updating additional view indexes.

    + + + +
    Sorting on reduce values +

    The sorting algorithm within the view system outputs information ordered by the generated + key within the view, and therefore it operates before any reduction takes place. + Unfortunately, it is not possible to sort the output order of the view on computed reduce + values, as there is no post-processing on the generated view information.

    To sort + based on reduce values, you must access the view content with reduction enabled from a + client, and perform the sorting within the client application.

    + + + +
    Solutions for simulating joins +

    Joins between data, even when the documents being examined are contained within the same + bucket, are not possible directly within the view system. However, you can simulate this by + making use of a common field used for linking when outputting the view information. For + example, consider a blog post system that supports two different record types, ‘blogpost’ + and ‘blogcomment’. The basic format for ‘blogpost’ + is:

    { + "type" : "post", + "title" : "Blog post" + "categories" : [...], + "author" : "Blog author" + ... +} +

    The + corresponding comment record includes the blog post ID within the document + structure:

    { + "type" : "comment", + "post_id" : "post_3454" + "author" : "Comment author", + "created_at" : 123498235 +... +} +

    To + output a blog post and all the comment records that relate to the blog post, you can use the + following + view:

    function(doc, meta) +{ + if (doc.post_id && doc.type && doc.type == "post") + { + emit([doc.post_id, null], null); + } + else if (doc.post_id && doc.created_at && doc.type && doc.type == "comment") + { + emit([doc.post_id, doc.created_at], null); + } +} +

    The + view makes use of the sorting algorithm when using arrays as the view key. For a blog post + record, the document ID will be output will a null second value in the array, and the blog + post record will therefore appear first in the sorted output from the view. For a comment + record, the first value will be the blog post ID, which will cause it to be sorted in line + with the corresponding parent post record, while the second value of the array is the date + the comment was created, allowing sorting of the child comments.

    For + example:

    {"rows":[ +{"key":["post_219",null], "value":{...}}, +{"key":["post_219",1239875435],"value":{...}}, +{"key":["post_219",1239875467],"value":{...}}, +] +} +

    Another + alternative is to make use of a multi-get operation within your client through the main + Couchbase SDK interface, which should load the data from cache. This lets you structure + your data with the blog post containing an array of the of the child comment records. For + example, the blog post structure might + be:

    { + "type" : "post", + "title" : "Blog post" + "categories" : [...], + "author" : "Blog author", + "comments": ["comment_2298","comment_457","comment_4857"], + ... +} +

    To + obtain the blog post information and the corresponding comments, create a view to find the + blog post record, and then make a second call within your client SDK to get all the comment + records from the Couchbase Server cache.

    + + + +
    Simulating transactions +

    Couchbase Server does not support transactions, but the effect can be simulated by writing + a suitable document and view definition that produces the effect while still only requiring + a single document update to be applied.

    For example, consider a typical banking + application, the document structure could be as + follows:

    { + "account" : "James", + "value" : 100 +} +

    A + corresponding record for another + account:

    { + "account" : "Alice", + "value" : 200 +} +

    To + get the balance of each account, the following map() + :

    function(doc, meta) { + if (doc.account && doc.value) + { + emit(doc.account,doc.value); + } +} +

    The + reduce() function can use the built-in _sum + function.

    When queried, using a group_level of 1, the balance of the + accounts is + displayed:

    {"rows":[ +{"key":"Alice","value":200}, +{"key":"James","value":100} +] +} +

    Money + in an account can be updated just by adding another record into the system with the account + name and value. For example, adding the + record:

    { + "account" : "James", + "value" : 50 +} +

    Re-querying + the view produces an updated balance for each + account:

    {"rows":[ +{"key":"Alice","value":200}, +{"key":"James","value":150} +] +} +

    However, + if Alice wants to transfer $100 to James, two record updates are required:

      +
    1. A record that records an update to Alice’s account to reduce the value by 100.
      2. +
    2. A record that records an update to James’s account to increase the value by + 100.
      3. +

    Unfortunately, the integrity of the transaction could be compromised in the event of a + problem between step 1 and step 2. Alice’s account may be deducted, without updates James’ + record.

    To simulate this operation while creating (or updating) only one record, a + combination of a transaction record and a view must be used. The transaction record looks + like + this:

    { + "fromacct" : "Alice", + "toacct" : "James", + "value" : 100 +} +

    The + above records the movement of money from one account to another. The view can now be updated + to handle a transaction record and output a row through emit() to update + the value for each + account.

    function(doc, meta) +{ + if (doc.fromacct) + { + emit(doc.fromacct, -doc.value); + emit(doc.toacct, doc.value); + } + else + { + emit(doc.account, doc.value); + } +} +

    The + above map() effectively generates two fake rows, one row subtracts the + amount from the source account, and adds the amount to the destination account. The + resulting view then uses the reduce() function to sum up the transaction + records for each account to arrive at a final + balance:

    {"rows":[ +{"key":"Alice","value":100}, +{"key":"James","value":250} +] +} +

    Throughout + the process, only one record has been created, and therefore transient problems with that + record update can be captured without corrupting or upsetting the existing stored + data.

    + + +
    Simulating multi-phase transactions +

    The technique in Simulating Transactions works if your data will allow the use of a view to + effectively roll-up the changes into a single operation. However, if your data and document + structure do not allow it then you can use a multi-phase transaction process to perform the + operation in a number of distinct stages.

    This method is not reliant on views, but the + document structure and update make it easy to find out if there are ‘hanging’ or trailing + transactions that need to be processed without additional document updates. Using views and + the Observe operation to monitor changes could lead to long wait times during the + transaction process while the view index is updated.

    To employ this method, you use a + similar transaction record as in the previous example, but use the transaction record to + record each stage of the update process.

    Start with the same two account + records:

    { + "type" : "account", + "account" : "James", + "value" : 100, + "transactions" : [] +} +

    The + record explicitly contains a transactions field which contains an array of + all the currently active transactions on this record.

    The corresponding record for the + other + account:

    { + "type" : "account", + "account" : "Alice", + "value" : 200, + "transactions" : [] +} +

    Now + perform the following operations in sequence:

    +
      +
    1. Create a new transaction record that records the transaction information:

      { + "type" : "transaction", "fromacct" : "Alice", "toacct" : "James", "value" : 100, + "status" : "waiting" }

      +

      The core of the transaction record is the same, + the difference is the use of a status field which will be used to + monitor the progress of the transaction.

      Record the ID of the transaction, for + example, transact_20120717163.

      2. +
    2. Set the value of the status field in the transaction document to + ‘pending’:

      { "type" : "transaction", "fromacct" : "Alice", "toacct" : + "James", "value" : 100, "status" : "pending" }

      3. +
    3. Find all transaction records in the pending state using a suitable + view:

      function(doc, meta) { if (doc.type && doc.status && + doc.type == "transaction" && doc.status == "pending" ) { + emit([doc.fromacct,doc.toacct], doc.value); } }

      4. +
    4. Update the record identified in toacct with the transaction + information, ensuring that the transaction is not already pending:

      { "type" : + "account", "account" : "Alice", "value" : 100, "transactions" : + ["transact_20120717163"] }

      Repeat on the other account:

      +

      + { "type" : "account", "account" : "James", "value" : 200, "transactions" : + ["transact_20120717163"] }

      5. +
    5. Update the transaction record to mark that the records have been updated:

      { + "type" : "transaction", "fromacct" : "Alice", "toacct" : "James", "value" : 100, + "status" : "committed" }

      6. +
    6. Find all transaction records in the committed state using a suitable + view:

      function(doc, meta) { if (doc.type && doc.status && + doc.type == "transaction" && doc.status == "committed" ) { emit([doc.fromacct, + doc.toacct], doc.value); } }

      Update the source account record noted in + the transaction and remove the transaction ID:

      { "type" : "account", + "account" : "Alice", "value" : 100, "transactions" : [] }

      Repeat on + the other account:

      { "type" : "account", "account" : "James", "value" : + 200, "transactions" : [] }

      7. +
    7. Update the transaction record state to ‘done’. This will remove the transaction from the + two views used to identify unapplied, or uncommitted transactions.
      8. +

    Within this process, although there are multiple steps required, you can identify at + each step whether a particular operation has taken place or not.

    For example, if the + transaction record is marked as ‘pending’, but the corresponding account records do not + contain the transaction ID, then the record still needs to be updated. Since the account + record can be updated using a single atomic operation, it is easy to determine if the record + has been updated or not.

    The result is that any sweep process that accesses the views + defined in each step can determine whether the record needs updating. Equally, if an + operation fails, a record of the transaction, and whether the update operation has been + applied, also exists, allowing the changes to be reversed and backed out.

    + +     diff --git a/content/understanding-couchbase/views/views-querying.dita b/content/understanding-couchbase/views/views-querying.dita new file mode 100644 index 0000000000..66117af0c0 --- /dev/null +++ b/content/understanding-couchbase/views/views-querying.dita @@ -0,0 +1,708 @@ + + + + Querying MapReduce Views + The content of the key that is generated by the emit() function + provides information on how the data is selected from your view. + +

    In order to query a view, the view definition must include a suitable map function that + uses the emit() function to generate each row of information.

    +

    The key can be used when querying a view as the selection mechanism, either by using + an:

    +
      +
    • +

      explicit key — show all the records matching the exact structure of the + supplied key.

      +
      • +
    • +

      list of keys — show all the records matching the exact structure of each of + the supplied keys (effectively showing keya or keyb or keyc).

      +
      • +
    • +

      range of keys — show all the records starting with keyA and stopping on the + last instance of keyB.

      +
      • +
    +

    When querying the view results, a number of parameters can be used to select, limit, order + and otherwise control the execution of the view and the information that is returned.

    +

    When a view is accessed without specifying any parameters, the view produces results + matching the following:

    +
      +
    • +

      Full view specification, i.e. all documents are potentially output according to the + view definition.

      +
      • +
    • +

      Limited to 10 items within the web console, unlimited through the REST API.

      +
      • +
    • +

      Reduce function used if defined in the view.

      +
      • +
    • +

      Items sorted in ascending order (using UTF-8 comparison for strings, natural number + order)

      +
      • +
    +

    View results and the parameters operate and interact in a specific order. The interaction + directly affects how queries are written and data is selected.

    + + + + + +

    The core arguments and selection systems are the same through both the REST API interface, + and the client libraries. The setting of these values differs between different client + libraries, but the argument names and expected and supported values are the same across all + environments.

    +
    Querying

    Querying can be performed through the REST API endpoint. + The REST API supports and operates using the core HTTP protocol, and this is the same + system used by the client libraries to obtain the view data.

    To retrieve views + information, access any server node in a cluster on port 8092.

    The following is the HTTP + method and URI used to query + views:

    GET /[bucket-name]/_design/[ddoc-name]/_view/[view-name]

    Where:

      +
    • bucket-name is the name of the bucket.
      • +
    • ddoc-name is the name of the design document that contains the view.
      • +
    • view-name is the name of the corresponding view within the design document.
      • +

    Development view, the ddoc-name is prefixed with + dev_. For example, the design document beer is + accessible as a development view using dev_beer.

    Production views + are accessible using their name only.

    +

    Parameters (optional):

    + + Views parameters + + + + + + + Parameters + Type + Description + + + + + descending + boolean + Return the documents in descending by key order. + + + endkey + string + Stop returning records when the specified key is reached. Key must be + specified as a JSON value. + + + endkey_docid + string + Stop returning records when the specified document ID is + reached. + + + full_set + boolean + Use the full cluster data set (development views only). + + + group + boolean + Group the results using the reduce function to a group or single row. + Note: Do not use group with group_level + because they are not compatible. + + + group_level + numeric + Specify the group level to be used. Note: Do not use + group_level with group because they are + not compatible. + + + inclusive_end + boolean + Specifies whether the specified end key is included in the result. Note: + Do not use inclusive_end with key or + keys. + + + key + string + Return only documents that match the specified key. Key must be + specified as a JSON value. + + + keys + array + Return only documents that match each of keys specified within the given + array. Key must be specified as a JSON value. Sorting is not applied when + using this option. + + + limit + numeric + Limit the number of the returned documents to the specified + number. + + + on_error + string + Sets the response in the event of an error.

    Supported values:

      +
    • continue : Continue to generate view information + in the event of an error, including the error information in the + view response stream.
      • +
    • stop : Stop immediately when an error condition + occurs. No further view information is returned.
      • +

    +     +     + + reduce + boolean + Use the reduction function. + + + skip + numeric + Skip this number of records before starting to return the + results. + + + stale + string + Allow the results from a stale view to be used.

    Supported values:

      +
    • false : The server waits for the indexer to + finish the changes that correspond to the current key-value + document set and then returns the latest entries from the view + index.
      • +
    • ok : The server returns the current entries from + the index file including the stale views.
      • +
    • update_after : The server returns the current + entries from the index, and then initiates an index update.
      • +

    +     + + startkey + string + Return records with a value equal to or greater than the specified key. + Key must be specified as a JSON value. + + + startkey_docid + string + Return records starting with the specified document ID. + + + +
    +

    Curl request syntax:

    + GET http://[localhost]:8092/[bucket-name]/_design/[ddoc-name]/_view/[view-name] +

    To access a view stored within an SASL password-protected bucket, include the bucket + name and bucket password within the URL of the request:

    + GET http://[bucket-name]:[password]@[localhost]:8092/[bucket-name]/_design/[ddoc-name]/_view/[view-name] + + Additional arguments to the URL request can be used to select information + from the view, and provide limit, sorting and other options. +

    To output only ten items:

    + GET http://[localhost]:8092/[bucket-name]/_design/[ddoc-name]/_view/[view-name]?limit=10 + + The formatting of the URL follows the HTTP specification. The first + argument is separated from the base URL using a question mark ( ? ). + Additional arguments are separated using an ampersand ( & ). + Special characters are quoted or escaped according to the HTTP standard rules. +
    +
    + Selecting information +

    Couchbase Server supports a number of mechanisms for selecting information returned by + the view. Key selection is made after the view results (including the reduction + function) are executed, and after the items in the view output have been sorted.

    +

    When specifying keys to the selection mechanism, the key must be expressed in the form + of a JSON value. For example, when specifying a single key, a string must be quoted + ("string").

    +

    When specifying the key selection through a parameter, the keys must match the format of + the keys emitted by the view. Compound keys, for example where an array or hash has been + used in the emitted key structure, the supplied selection value should also be an array + or a hash.

    +

    The following selection types are supported:

    +
      +
    • Explicit Key
      • +
    +

    An explicit key can be specified using the parameter key. The view + query will only return results where the key in the view output, and the value supplied + to the key parameter match identically.

    +

    For example, if you supply the value "tomato" only records matching exactly + "tomato" will be selected and returned. Keys with values such as "tomatoes" will not be + returned.

    +
      +
    • Key List
      • +
    +

    A list of keys to be output can be specified by supplying an array of values using the + keys parameter. In this instance, each item in the specified array + will be used as explicit match to the view result key, with each array value being + combined with a logical or.

    +

    For example, if the value specified to the keys parameter was + ["tomato","avocado"], then all results with a key of ‘tomato’ + or ‘avocado’ will be returned.

    +

    When using this query option, the output results are not sorted by key. This is because + key sorting of these values would require collating and sorting all the rows before + returning the requested information.

    +

    In the event of using a compound key, each compound key must be specified in the query. + For example:

    + ``` +keys=[["tomato",20],["avocado",20]] +``` + +
      +
    • Key Range
      • +
    +

    A key range, consisting of a startkey and endkey. + These options can be used individually, or together, as follows:

    + * `startkey` only + + Output does not start until the first occurrence of `startkey`, or a value + greater than the specified value, is seen. Output will then continue until the + end of the view. + +* `endkey` only + + Output starts with the first view result, and continues until the last + occurrence of `endkey`, or until the emitted value is greater than the computed + lexical value of `endkey`. + +* `startkey` and `endkey` + + Output of values does not start until `startkey` is seen, and stops when the + last occurrence of `endkey` is identified. + +

    When using endkey, the inclusive_end option specifies + whether output stops after the last occurrence of the specified endkey + (the default). If set to false, output stops on the last result before the specified + endkey is seen.

    +

    The matching algorithm works on partial values, which can be used to an advantage when + searching for ranges of keys.

    + Do not use the inclusive_end parameter with + key or keys parameters. The + inclusive_end parameter is not designed to work with + key or keys because it is an attribute of range + operations. +
    +
    + Selecting compound information by key or keys +

    If you are generating a compound key within your view, for example when outputting a + date split into individually year, month, day elements, then the selection value must + exactly match the format and size of your compound key. The value of + key or keys must exactly match the output key + structure.

    +

    For example, with the view data:

    + {"total_rows":5693,"rows":[ +{"id":"1310653019.12667","key":[2011,7,14,14,16,59],"value":null}, +{"id":"1310662045.29534","key":[2011,7,14,16,47,25],"value":null}, +{"id":"1310668923.16667","key":[2011,7,14,18,42,3],"value":null}, +{"id":"1310675373.9877","key":[2011,7,14,20,29,33],"value":null}, +{"id":"1310684917.60772","key":[2011,7,14,23,8,37],"value":null}, +{"id":"1310693478.30841","key":[2011,7,15,1,31,18],"value":null}, +{"id":"1310694625.02857","key":[2011,7,15,1,50,25],"value":null}, +{"id":"1310705375.53361","key":[2011,7,15,4,49,35],"value":null}, +{"id":"1310715999.09958","key":[2011,7,15,7,46,39],"value":null}, +{"id":"1310716023.73212","key":[2011,7,15,7,47,3],"value":null} +] +} + +

    Using the key selection mechanism you must specify the entire key + value, i.e.:

    + ?key=[2011,7,15,7,47,3] + +

    If you specify a value, such as only the date:

    + ?key=[2011,7,15] + +

    The view will return no records, since there is no exact key match. Instead, you must + use a range that encompasses the information range you want to output:

    + ?startkey=[2011,7,15,0,0,0]&endkey=[2011,7,15,99,99,99] + +

    This will output all records within the specified range for the specified date.

    +
    +
    + Partial selection and key ranges +

    Matching of the key value has a precedence from right to left for the key value and the + supplied startkey and/or endkey. Partial strings may + therefore be specified and return specific information.

    +

    For example, given the view data:

    + "a", + "aa", + "bb", + "bbb", + "c", + "cc", + "ccc" + "dddd" + +

    Specifying a startkey parameter with the value "aa" will return the + last seven records, including "aa":

    + "aa", + "bb", + "bbb", + "c", + "cc", + "ccc", + "dddd" + +

    Specifying a partial string to startkey will trigger output of the + selected values as soon as the first value or value greater than the specified value is + identified. For strings, this partial match (from left to right) is identified. For + example, specifying a startkey of "d" will return:

    + "dddd" + +

    This is because the first match is identified as soon as the a key from a view row + matches the supplied startkey value from left to right. The + supplied single character matches the first character of the view output.

    +

    When comparing larger strings and compound values the same matching algorithm is used. + For example, searching a database of ingredients and specifying a + startkey of "almond" will return all the ingredients, including + "almond", "almonds", and "almond essence".

    +

    To match all of the records for a given word or value across the entire range, you can + use the null value in the endkey parameter. For example, to search for + all records that start only with the word "almond", you specify a + startkey of "almond", and an endkey of "almond\u02ad" (i.e. with the + last Latin character at the end). If you are using Unicode strings, you may want to use + "\uefff".

    + startkey="almond"&endkey="almond\u02ad" + +

    The precedence in this example is that output starts when ‘almond’ is seen, and stops + when the emitted data is lexically greater than the supplied endkey. + Although a record with the value "almond\02ad" will never be seen, the emitted data will + eventually be lexically greater than "almond\02ad" and output will stop.

    +

    In effect, a range specified in this way acts as a prefix with all the data being output + that match the specified prefix.

    +
    +
    + Partial selection with compound keys +

    Compound keys, such as arrays or hashes, can also be specified in the view output, and + the matching precedence can be used to provide complex selection ranges. For example, if + time data is emitted in the following format:

    + [year,month,day,hour,minute] + +

    Then precise date (and time) ranges can be selected by specifying the date and time in + the generated data. For example, to get information between 1st April 2011, 00:00 and + 30th September 2011, 23:59:

    + ?startkey=[2011,4,1,0,0]&endkey=[2011,9,30,23,59] + +

    The flexible structure and nature of the startkey and + endkey values enable selection through a variety of range + specifications. For example, you can obtain all of the data from the beginning of the + year until the 5th March using:

    + ?startkey=[2011]&endkey=[2011,3,5,23,59] + +

    You can also examine data from a specific date through to the end of the month:

    + ?startkey=[2011,3,16]&endkey=[2011,3,99] + +

    In the above example, the value for the day element of the array is an + impossible value, but the matching algorithm will identify when the emitted value is + lexically greater than the supplied endkey value, and information + selected for output will be stopped.

    +

    A limitation of this structure is that it is not possible to ignore the earlier array + values. For example, to select information from 10am to 2pm each day, you cannot use + this parameter set:

    + ?startkey=[null,null,null,10,0]&endkey=[null,null,null,14,0] + +

    In addition, because selection is made by a outputting a range of values based on the + start and end key, you cannot specify range values for the date portion of the + query:

    + ?startkey=[0,0,0,10,0]&endkey=[9999,99,99,14,0] + +

    This will instead output all the values from the first day at 10am to the last day at + 2pm.

    +
    +
    + Pagination +

    Pagination over results can be achieved by using the skip and + limit parameters. For example, to get the first 10 records from the + view:

    + ?limit=10 + +

    The next ten records can obtained by specifying:

    + ?skip=10&limit=10 + +

    On the server, the skip option works by executing the query and + literally iterating over the specified number of output records specified by + skip, then returning the remainder of the data up until the + specified limit records are reached, if the limit + parameter is specified.

    +

    When paginating with larger values for skip, the overhead for iterating + over the records can be significant. A better solution is to track the document id + output by the first query (with the limit parameter). You can then use + startkey_docid to specify the last document ID seen, skip over that + record, and output the next ten records.

    +

    Therefore, the paging sequence is, for the first query:

    + ?startkey="carrots"&limit=10 + +

    Record the last document ID in the generated output, then use:

    + ?startkey="carrots"&startkey_docid=DOCID&skip=1&limit=10 + +

    When using startkey_docid you must specify the + startkey parameter to specify the information being searched for. By + using the startkey_docid parameter, Couchbase Server skips through the + B-Tree index to the specified document ID. This is much faster than the skip/limit + example shown above.

    +
    +
    + Grouping in queries +

    If you have specified an array as your compound key within your view, then you can + specify the group level to be applied to the query output when using a + reduce().

    +

    When grouping is enabled, the view output is grouped according to the key array, and you + can specify the level within the defined array that the information is grouped by. You + do this by specifying the index within the array by which you want the output grouped + using the group_level parameter.

    + + + + + +

    The group_level parameter specifies the array index (starting at 1) at + which you want the grouping occur, and generate a unique value based on this value that + is used to identify all the items in the view output that include this unique value:

    +
      +
    • +

      A group level of 0 groups by the entire dataset (as if no array + exists).

      +
      • +
    • +

      A group level of 1 groups the content by the unique value of the + first element in the view key array. For example, when outputting a date split by + year, month, day, hour, minute, each unique year will be output.

      +
      • +
    • +

      A group level of 2 groups the content by the unique value of the + first and second elements in the array. With a date, this outputs each unique year + and month, including all records with that year and month into each group.

      +
      • +
    • +

      A group level of 3 groups the content by the unique value of the + first three elements of the view key array. In a date this outputs each unique + date (year, month, day) grouping all items according to these first three + elements.

      +
      • +
    +

    The grouping will work for any output structure where you have output an compound key + using an array as the output value for the key.

    +
    +
    + Selection when grouping +

    When using grouping and selection using the key, keys, + or startkey / endkey parameters, the query value + should match at least the format (and element count) of the group level that is being + queried.

    +

    For example, using the following map() function to output information + by date as an array:

    + function(doc, meta) +{ + emit([doc.year, doc.mon, doc.day], doc.logtype); +} + +

    If you specify a group_level of 2 then you must + specify a key using at least the year and month information. For example, you can + specify an explicit key, such as [2012,8] :

    + ?group_level=2&key=[2012,8] +

    You can query it for a range:

    + ?group_level=2&startkey=[2012,2]&endkey=[2012,8] + +

    You can also specify a year, month and day, while still grouping at a higher level. For + example, to group by year/month while selecting by specific dates:

    + ?group_level=2&startkey=[2012,2,15]&endkey=[2012,8,10] + +

    Specifying compound keys that are shorter than the specified group level may output + unexpected results due to the selection mechanism and the way startkey + and endkey are used to start and stop the selection of output rows.

    +
    +
    + Ordering +

    All view results are automatically output sorted, with the sorting based on the content + of the key in the output view. Views are sorted using a specific sorting format, with + the basic order for all basic and compound follows as follows:

    +
      +
    • +

      null

      +
      • +
    • +

      false

      +
      • +
    • +

      true

      +
      • +
    • +

      Numbers

      +
      • +
    • +

      Text (case sensitive, lowercase first, UTF-8 order)

      +
      • +
    • +

      Arrays (according to the values of each element, in order)

      +
      • +
    • +

      Objects (according to the values of keys, in key order)

      +
      • +
    +

    The natural sorting is therefore by default close to natural sorting order both + alphabetically (A-Z) and numerically (0-9).

    +

    There is no collation or foreign language support. Sorting is always according to the + above rules based on UTF-8 values.

    +

    You can alter the direction of the sorting (reverse, highest to lowest numerically, Z-A + alphabetically) by using the descending option. When set to true, this + reverses the order of the view results, ordered by their key.

    +

    Because selection is made after sorting the view results, if you configure the results + to be sorted in descending order and you are selecting information using a key range, + then you must also reverse the startkey and endkey + parameters. For example, if you query ingredients where the start key is ‘tomato’ and + the end key is ‘zucchini’, for example:

    + ?startkey="tomato"&endkey="zucchini" + +

    The selection will operate, returning information when the first key matches ‘tomato’ + and stopping on the last key that matches ‘zucchini’.

    +

    If the return order is reversed:

    + ?descending=true&startkey="tomato"&endkey="zucchini" + +

    The query will return only entries matching ‘tomato’. This is because the order will be + reversed, ‘zucchini’ will appear first, and it is only when the results contain ‘tomato’ + that any information is returned.

    +

    To get all the entries that match, the startkey and + endkey values must also be reversed:

    + ?descending=true&startkey="zucchini"&endkey="tomato" + +

    The above selection will start generating results when ‘zucchini’ is identified in the + key, and stop returning results when ‘tomato’ is identified in the key.

    +

    View output and selection are case sensitive. Specifying the key ‘Apple’ will not return + ‘apple’ or ‘APPLE’ or other case differences. Normalizing the view output and query + input to all lowercase or upper case will simplify the process by eliminating the case + differences.

    +
    +
    + Understanding letter ordering in views +

    Couchbase Server uses a Unicode collation algorithm to order letters, so you should be + aware of how this functions. Most developers are typically used to Byte order, such as + that found in ASCII and which is used in most programming languages for ordering strings + during string comparisons.

    +

    The following shows the order of precedence used in Byte order, such as ASCII:

    + 123456890 < A-Z < a-z + +

    This means any items that start with integers will appear before any items with letters; + any items that beginning with capital letters will appear before items in lower case + letters. This means the item named "Apple" will appear before "apple" and the item + "Zebra" will appear before "apple". Compare this with the order of precedence used in + Unicode collation, which is used in Couchbase Server:

    + 123456790 < aAbBcCdDeEfFgGhH... + +

    Notice again that items that start with integers will appear before any items with + letters. However, in this case, the lowercase and then uppercase of the same letter are + grouped together. This means that that if "apple" will appear before "Apple" and would + also appear before "Zebra." In addition, be aware that with accented characters will + follow this ordering:

    + a < á < A < Á < b + +

    This means that all items starting with "a" and accented variants of the letter + will occur before "A" and any accented variants of "A."

    +

    Ordering Example

    +

    In Byte order, keys in an index would appear as follows:

    + "ABC123" < "ABC223" < "abc123" < "abc223" < "abcd23" < "bbc123" < "bbcd23" + +

    The same items will be ordered this way by Couchbase Server under Unicode collation:

    + "abc123" < "ABC123" < "abc223" < "ABC223" < "abcd23" < "bbc123" < "bbcd23" + +

    This is particularly important for you to understand if you query Couchbase Server with + a startkey and endkey to get back a range of results. + The items you would retrieve under Byte order are different compared to Unicode + collation.

    +

    Ordering and Query Example

    +

    This following example demonstrates Unicode collation in Couchbase Server and the impact + on query results returned with a startkey and endkey. + It is based on the beer-sample database provided with Couchbase Server.

    +

    Imagine you want to retrieve all breweries with names starting with uppercase Y. Your + query parameters would appear as follows:

    + startkey="Y"&endkey="z" + +

    If you want breweries starting with lowercase y or uppercase Y, you would + provides a query as follows:

    + startkey="y"&endkey="z" + +

    This will return all names with lower case Y and items up to, but not including + lowercase z, thereby including uppercase Y as well. To retrieve the names of breweries + starting with lowercase y only, you would terminate your range with capital Y:

    + startkey="y"&endkey="Y" + +

    As it happens, the sample database does not contain any results because there are no + beers in it which start with lowercase y.

    +
    +
    + Error control +

    There are a number of parameters that can be used to help control errors and responses + during a view query.

    +
      +
    • on_error
      • +
    +

    The on_error parameter specifies whether the view results will be + terminated on the first error from a node, or whether individual nodes can fail and + other nodes return information.

    +

    When returning the information generated by a view request, the default response is for + any raised error to be included as part of the JSON response, but for the view process + to continue. This permits for individual nodes within the Couchbase cluster to timeout + or fail, while still generating the requested view information.

    +

    In this instance, the error is included as part of the JSON returned:

    + +{ + "errors" : [ + { + "from" : "http://192.168.1.80:9503/_view_merge/?stale=false", + "reason" : "req_timedout" + }, + { + "from" : "http://192.168.1.80:9502/_view_merge/?stale=false", + "reason" : "req_timedout" + }, + { + "from" : "http://192.168.1.80:9501/_view_merge/?stale=false", + "reason" : "req_timedout" + } + ], + "rows" : [ + { + "value" : 333280, + "key" : null + } + ] +} + +

    You can alter this behavior by using the on_error argument. The default + value is continue. If you set this value to stop then + the view response will cease the moment an error occurs. The returned JSON will contain + the error information for the node that returned the first error. For example:

    + ``` +{ + "errors" : [ + { + "from" : "http://192.168.1.80:9501/_view_merge/?stale=false", + "reason" : "req_timedout" + } + ], + "rows" : [ + { + "value" : 333280, + "key" : null + } + ] +} +``` + +
    + + + + + Unicode Technical Standard #10 + + + ICU User Guide, Customization, Default Options + + + +     diff --git a/content/understanding-couchbase/views/views-schemaless.dita b/content/understanding-couchbase/views/views-schemaless.dita new file mode 100644 index 0000000000..9a3972465b --- /dev/null +++ b/content/understanding-couchbase/views/views-schemaless.dita @@ -0,0 +1,52 @@ + + + + Views in a Schema-less Database + A schema-less database along with view definitions provide for a flexible document structure. + +

    One of the primary advantages of the document-based storage and the use of map/reduce views + for querying the data is that the structure of the stored documents does not need to be + predeclared, or even consistent across multiple documents.

    +

    Instead, the view can cope with and determine the structure of the incoming documents that + are stored in the database, and the view can then reformat and restructure this data during + the map/reduce stage. This simplifies the storage of information, both in the initial format, + and over time, as the format and structure of the documents can change over time.

    +

    For example, you could start storing name information using the following JSON structure:

    + { + "email" : "mc@example.org", + "name" : "Martin Brown" +} + +

    A view can be defined that outputs the email and name:

    + function(doc, meta) +{ + emit([doc.name, doc.email], null); +} +

    This generates an index containing the name and email information. Over time, the application + is adjusted to store the first and last names separately:

    + { + "email" : "mc@example.org", + "firstname" : "Martin", + "lastname" : "Brown" +} + +

    The view can be modified to cope with both the older and newer document types, while still + emitting a consistent view:

    + function(doc, meta) +{ + if (doc.name && (doc.name != null)) + { + emit([doc.name, doc.email], null); + } + else + { + emit([doc.firstname + " " + doc.lastname, doc.email], null); + } +} + +

    The schema-less nature and view definitions provide for a flexible document structure, and an + evolving one, without requiring either an initial schema description, or explicit schema + updates when the format of the information changes.

    + +     diff --git a/content/understanding-couchbase/views/views-store-data.dita b/content/understanding-couchbase/views/views-store-data.dita new file mode 100644 index 0000000000..5d2c93ea59 --- /dev/null +++ b/content/understanding-couchbase/views/views-store-data.dita @@ -0,0 +1,179 @@ + + + + Views and Stored Data + This section describes how the views system relates to stored data. + +

    The view system relies on the information being formatted as a JSON document. The formatting + of the data in this form enables the individual fields of the data to be identified and + used.

    +

    Information is stored into your Couchbase database the data stored is parsed, if the + information can be identified as valid JSON then the information is tagged and identified in + the database as valid JSON. If the information cannot be parsed as valid JSON then it is + stored as a verbatim binary copy of the submitted data.

    + + + +

    When retrieving the stored data, the format of the information depends on whether the data + was tagged as valid JSON or not:

    + +
    JSON information + +

    Information identified as JSON data may not be returned in a format identical to that stored. + The information will be semantically identical, in that the same fields, data and structure as + submitted will be returned. Metadata information about the document is presented in a separate + structure available during view processing.

    +

    The white space, field ordering may differ from the submitted version of the JSON + document.

    +

    For example, the JSON document below, stored using the key mykey :

    + + +{ + "title" : "Fish Stew", + "servings" : 4, + "subtitle" : "Delicious with fresh bread" +} + + +

    May be returned within the view processor as:

    + +{ + "servings": 4, + "subtitle": "Delicious with fresh bread", + "title": "Fish Stew" +} + +
    + +
    Non-JSON information + +

    Information not parsable as JSON will always be stored and returned as a binary copy of the + information submitted to the database. If you store an image, for example, the data returned + will be an identical binary copy of the stored image.

    +

    Non-JSON data is available as a base64 string during view processing. A non-JSON document can + be identified by examining the type field of the metadata structure.

    +

    The significance of the returned structure can be seen when editing the view within the Web + Console.

    +
    + + +
    JSON basics +

    JSON is used because it is a lightweight, easily parsed, + cross-platform data representation format. There are a multitude of libraries and tools + designed to help developers work efficiently with data represented in JSON format, on every + platform and every conceivable language and application framework, including, of course, + most web browsers.

    +

    JSON supports the same basic types as supported by JavaScript, + these are:

    +
      +
    • Number (either integer or floating-point).
      • +

    JavaScript supports a maximum numerical value of 253. If you are working + with numbers larger than this from within your client library environment (for example, + 64-bit numbers), you must store the value as a string.

      +

    • String — this should be enclosed by double-quotes and supports Unicode characters and + backslash escaping. For example:

      + "A String"
      • +

    • Boolean — a true or false value. You can use these + strings directly. For example:

      { "value": true}
      • +

    • Array — a list of values enclosed in square brackets. For example:

      + ["one", "two", "three"]
      • +

    • Object — a set of key-value pairs (such as an associative array or hash). The key must + be a string, but the value can be any of the supported JSON values. For example:

      + { + "servings" : 4, + "subtitle" : "Easy to make in advance, and then cook when ready", + "cooktime" : 60, + "title" : "Chicken Coriander" +} +
      • +
    +

    If the submitted data cannot be parsed as a JSON, the information will be stored as a + binary object, not a JSON document.

    + + +
    Document metadata +

    During view processing, metadata about individual + documents is exposed through a separate JSON object, meta, that can be + optionally defined as the second argument to the map(). This metadata can + be used to further identify and qualify the document being processed.

    +

    The + meta structure contains the following fields and associated + information:

    +
      +
    • id
      • +

    The ID or key of the stored data object. This is the same as the key used when writing + the object to the Couchbase database.

      +
    • rev
      • +

    An internal revision ID used internally to track the current revision of the + information. The information contained within this field is not consistent or trackable and + should not be used in client applications.

      +
    • type
      • +

    The type of the data that has been stored. A valid JSON document will have the type + json. Documents identified as binary data will have the type + base64.

      +
    • flags
      • +

    The numerical value of the flags set when the data was stored. The availability and + value of the flags is dependent on the client library you are using to store your data. + Internally the flags are stored as a 32-bit integer.

      +
    • expiration
      • +
    +

    The expiration value for the stored object. The stored expiration time is always + stored as an absolute Unix epoch time value.

    These additional fields are only exposed + when processing the documents within the view server. These fields are not returned when you + access the object through the Memcached/Couchbase protocol as part of the + document.

    +
    + + +
    Non-JSON data

    All documents stored in Couchbase Server will return a + JSON structure, however, only submitted information that could be parsed into a JSON + document will be stored as a JSON document. If you store a value that cannot be parsed as a + JSON document, the original binary data is stored. This can be identified during view + processing by using the meta object supplied to the map() + function.

    Information that has been identified and stored as binary documents instead + of JSON documents can still be indexed through the views system by creating an index on the + key data. This can be particularly useful when the document key is significant. For example, + if you store information using a prefix to the key to identify the record type, you can + create document-type specific indexes.

    + + +
    Document storage and indexing sequence

    The method of storage of + information into the Couchbase Server affects how and when the indexing information is + built, and when data written to the cluster is incorporated into the indexes. In addition, + the indexing of data is also affected by the view system and the settings used when the view + is accessed.

    + + + + +

    The basic storage and indexing sequence is:

    + + + +
      +
    1. A document is stored within the cluster. Initially the document is stored only in RAM.
      2. +
    2. The document is communicated to the indexer through replication to be indexed by views.
      3. +
    + +

    This sequence means that the view results are eventually consistent with what is stored in + memory based on the latency in replication of the change to the indexer. It is possible to + write a document to the cluster and access the index without the newly written document + appearing in the generated view.

    + + +

    Conversely, documents that have been stored with an expiry may continue to be included + within the view until the document has been removed from the database by the expiry pager.

    + +

    Couchbase Server supports the Observe command, which enables the current state of a document and + whether the document has been replicated to the indexer or whether it has been considered for inclusion in an index. +

    +

    When accessing a view, the contents of the view are asynchronous to the stored documents. + In addition, the creation and updating of the view is subject to the parameter. + This controls how and when the view is updated when the view content is queried. +

    +
    + + + +     diff --git a/content/understanding-couchbase/views/views-streaming.dita b/content/understanding-couchbase/views/views-streaming.dita new file mode 100644 index 0000000000..35cbd6ef2b --- /dev/null +++ b/content/understanding-couchbase/views/views-streaming.dita @@ -0,0 +1,64 @@ + + + + + Stream-based Views + + Stream-based views reduce latency for view updates, providing faster view + consistency and fresher data.Views use the streaming protocol, Database + Change Protocol (DCP). + + + + +

    With DCP, data does not need to be persisted to disk before retrieving it with a view query. + DCP offers the following benefits for views:

    +
      +
    • Views are updated with key-value data sooner.
      • +
    • Latency for view updates is reduced.
      • +
    • Views are more closely synchronized with the data.
      • +
    + +

    The following diagram shows, at a high-level, how stream-based views operate.

    + + + +

    When submitting a view query, a parameter can be included that specifies data freshness + requirements. The name of the parameter is stale and it takes the + following values:

    + +
      +
    • ok—The server returns the current entries from the index file.
      • +
    • update_after—The server returns the current entries from the index, + and then initiates an index update.
      • +
    • false—The server waits for the indexer to finish the changes that + correspond to the current key-value document set and then returns the latest entries + from the view index.
      • +
    + +

    Every 5 seconds the automatic update process checks whether 5000 changes have occurred. If + a minimum of 5000 changes occurred, an index update is triggered. Otherwise, no update is + triggered. When triggered, the indexer requests from DCP all changes since it was last run. + The default number of changes to check for is 5000, but that number can be configured by + setting the updateMinChanges option. The update interval can also be + configured by setting the updateInterval option.

    + +

    The stale=false view query argument has been enhanced. When an application + sends a query that has the stale parameter set to false, the application + receives all recent changes to the documents, including changes that haven't yet been + persisted to disk. It considers all document changes that have been received at the time + the query was received. This means that using the durability requirements or observe + feature to block for persistence in application code before issuing the + stale=false query is no longer needed. It is recommended that you + remove all such application level checks after upgrading.

    + + + For better scalability and throughput, we recommend + that you set the value of the stale parameter to ok. + With the stream-based views, data returned when stale is set to + ok is closer to the key-value data, even though it might not + include all of it. + +     + +     diff --git a/content/understanding-couchbase/views/views-trans-sql.dita b/content/understanding-couchbase/views/views-trans-sql.dita new file mode 100644 index 0000000000..ddcc8f0b56 --- /dev/null +++ b/content/understanding-couchbase/views/views-trans-sql.dita @@ -0,0 +1,403 @@ + + + + Translating SQL to MapReduce + This section provides information on how to translate SQL to a MapReduce + environment. + + +

    If you have existing SQL queries and need materialized reductions, you can follow the + guidelines on this page for translating SQL queries to MapReduce views.

    + + If you don't need materialized reductions, we recommend that you + reformulate your SQL queries to use N1QL rather than MapReduce views. + + + +

    Here's an example of a SQL statement that you might want to translate:

    + + SELECT fieldlist FROM table + WHERE condition + GROUP BY groupfield + ORDER BY orderfield + LIMIT limitcount OFFSET offsetcount + +

    The different elements within the SQL statement affect how a view is written. The following + table describes how each clause is handled in a MapReduce view:

    + + + + + + + SQL clause + MapReduce + + + + + SELECT fieldlist + The field list within the SQL statement affects either the corresponding key or + value within the map() function, depending on whether you are also + selecting or reducing your data. + + + FROM table + There are no table compartments within Couchbase Server and you cannot perform + views across more than one bucket boundary. However, if you are using a + type field within your documents to identify different record + types, then you may want to use the map() function to make a + selection. + + + WHERE condition + The map() function and the data generated into the view key + directly affect how you can query, and therefore how selection of records takes + place. + + + GROUP BY groupfield + Grouping within SQL is handled within views through the use of the + reduce() function. + + + ORDER BY orderfield + The order of record output within a view is directly controlled by the key + specified during the map() function phase of the view + generation. + + + LIMIT limitcount OFFSET offsetcount + There are a number of different paging strategies available within the MapReduce + and views mechanism. + + + +
    + +

    View map() function, reduce() + function, selection parameters and other miscellaneous parameters have the following interactions:

    + + + + + + + + + + + + + + SQL clause + View key + View value + map() function + reduce() function + Selection parameters + Other parameters + + + + + SELECT fields + Yes + Yes + Yes + No: with GROUP BY and SUM() or + COUNT() functions only + No + No + + + FROM table + No + No + Yes + No + No + No + + + WHERE clause + Yes + No + Yes + No + Yes + No + + + ORDER BY field + Yes + No + Yes + No + No + descending + + + LIMIT x OFFSET y + No + No + No + No + No + limit, skip + + + GROUP BY field + Yes + Yes + Yes + Yes + No + No + + + +
    +

    Within SQL, the basic query structure can be used for a multitude of different queries. For + example, the structure SELECT fieldlist FROM table WHERE xxxx can be used with a + number of different clauses.

    +

    Within MapReduce and Couchbase Server, you might need to create multiple views to handle different types of queries. For example, performing a query on all the blog posts on a specific + date needs a very different view definition than one needed to support selection by the + author.

    + + + + +
    Translating SQL SELECT to MapReduce

    The field selection within an SQL + query can be translated into a corresponding view definition by:

    +
      +
    • Adding the fields to the emitted key if the value is also used for selection in a WHERE + clause.
      • +
    • Into the emitted value if the data is separate from the required query parameters.
      • +
    +

    For example, to get the sales data by country from each stored document + using the following map() + function:

    function(doc, meta) { + emit([doc.city, doc.sales], null); +} +

    If + you want to output information that can be used within a reduce function, this should be + specified in the value generated by each emit() call. For example, to + reduce the sales figures the above map() function could be rewritten + as:

    function(doc, meta) { + emit(doc.city, doc.sales); +} +

    In + essence this does not produce significantly different output (albeit with a simplified key), + but the information can now be reduced using the numerical value.

    If you want to + output data or field values completely separate to the query values, then these fields can + be explicitly output within the value portion of the view. For + example:

    function(doc, meta) { + emit(doc.city, [doc.name, doc.sales]); +} +

    If + the entire document for each item is required, load the document data after the view has + been requested through the client library. For more information on this parameter and the + performance impact.

    Within a SELECT statement it is + common practice to include the primary key for a given record in the output. Within a view + this is not normally required, since the document ID that generated each row is always + included within the view + output.

    + + + +
    Translating SQL WHERE to MapReduce

    The WHERE clause + within an SQL statement forms the selection criteria for choosing individual records. Within + a view, the ability to query the data is controlled by the content and structure of the + key generated by the map() function.

    In general, for each WHERE clause you need to include the corresponding field in + the key of the generated view, and then use the key, + keys or startkey / endkey + combinations to indicate the data you want to select. The complexity occurs when you + need to perform queries on multiple fields. There are a number of different strategies + that you can use for this.

    The simplest way + is to decide whether you want to be able to select a specific combination, or whether you + want to perform range or multiple selections. For example, using our recipe database, if you + want to select recipes that use the ingredient ‘carrot’ and have a cooking time of exactly + 20 minutes, then you can specify these two fields in the map() + function:

    function(doc, meta) +{ + if (doc.ingredients) + { + for(i=0; i < doc.ingredients.length; i++) + { + emit([doc.ingredients[i].ingredient, doc.totaltime], null); + } + } +} +

    Then + the query is an array of the two selection + values:

    ?key=["carrot",20] +

    This is equivalent + to the SQL + query:

    SELECT recipeid FROM recipe JOIN ingredients on ingredients.recipeid = recipe.recipeid + WHERE ingredient = 'carrot' AND totaltime = 20 +

    If, + however, you want to perform a query that selects recipes containing carrots that can be + prepared in less than 20 minutes, a range query is possible with the same + map() + function:

    ?startkey=["carrot",0]&endkey=["carrot",20] +

    This works because of the sorting mechanism in a view, which outputs the information sequentially + with carrots first and a sequential number.

    More complex queries though are more difficult. What if you want to select + recipes with carrots and rice, still preparable in under 20 minutes?

    A standard map() function like that above wont work. A range query on both + ingredients will list all the ingredients between the two. There are a number of + solutions available to you. First, the easiest way to handle the timing selection is to + create a view that explicitly selects recipes prepared within the specified time. Here's + an example:

    function(doc, meta) +{ + if (doc.totaltime <= 20) + { + ... + } +} +

    Although this approach seems to severely limit your queries, remember you can create multiple + views, so you could create one for 10 minutes, one for 20 minutes, one for 30 minutes, + or whatever intervals you select. It’s unlikely that anyone will really want to select + recipes that can be prepared in 17 minutes, so such granular selection is overkill.

    The multiple ingredients is more + difficult to solve. One way is to use the client to perform two queries and merge the data. + For example, the map() + function:

    function(doc, meta) +{ + if (doc.totaltime &amp;&amp; doc.totaltime <= 20) + { + if (doc.ingredients) + { + for(i=0; i < doc.ingredients.length; i++) + { + emit(doc.ingredients[i].ingredient, null); + } + } + } +} +

    Two + queries, one for each ingredient can easily be merged by performing a comparison and count + on the document ID output by each view.

    The alternative is to output the ingredients + twice within a nested loop, like + this:

    function(doc, meta) +{ + if (doc.totaltime &amp;&amp; doc.totaltime <= 20) + { + if (doc.ingredients) + { + for (i=0; i < doc.ingredients.length; i++) + { + for (j=0; j < doc.ingredients.length; j++) + { + emit([doc.ingredients[i].ingredient, doc.ingredients[j].ingredient], null); + } + } + } + } +} +

    Now + you can perform an explicit query on both + ingredients:

    ?key=["carrot","rice"] +

    If you + really want to support flexible cooking times, then you can also add the cooking + time:

    function(doc, meta) +{ + if (doc.ingredients) + { + for (i=0; i < doc.ingredients.length; i++) + { + for (j=0; j < doc.ingredients.length; j++) + { + emit([doc.ingredients[i].ingredient, doc.ingredients[j].ingredient, recipe.totaltime], null); + } + } + } +} +

    And + now you can support a ranged query on the cooking time with the two ingredient + selection:

    ?startkey=["carrot","rice",0]&key=["carrot","rice",20] +

    This + would be equivalent + to:

    SELECT recipeid FROM recipe +JOIN ingredients i1 ON ingredients.recipeid = recipe.recipeid +JOIN ingredients i2 ON ingredients.recipeid = recipe.recipeid +WHERE (i1.ingredient IN ('carrot',rice')) AND +(i2.ingredient IN ('carrot',rice')) AND +(totaltime < 20 AND totaltime > 0) +
    + + + +
    Translating SQL ORDER BY to MapReduce

    The ORDER BY + clause within SQL controls the order of the records that are output. Ordering within a view + is controlled by the value of the key. However, the key also controls and supports the + querying mechanism.

    In SELECT statements where there is no explicit + WHERE clause, the emitted key can entirely support the sorting you want. + For example, to sort by the city and salesman name, the following map() + will achieve the required + sorting:

    function(doc, meta) +{ + emit([doc.city, doc.name], null) +} +

    If + you need to query on a value, and that query specification is part of the order sequence + then you can use the format above. For example, if the query basis is city, then you can + extract all the records for ‘London’ using the above view and a suitable range + query:

    ?endkey=["London\u0fff"]&startkey=["London"] +

    However, + if you want to query the view by the salesman name, you need to reverse the field order in + the emit() + statement:

    function(doc, meta) +{ + emit([doc.name,doc.city],null) +} +

    Now + you can search for a name while still getting the information in city order.

    The order + the output can be reversed (equivalent to ORDER BY field DESC ) by using + the descending query parameter.

    + + + +
    Translating SQL GROUP BY to MapReduce

    The GROUP BY + parameter within SQL provides summary information for a group of matching records according + to the specified fields, often for use with a numeric field for a sum or total value, or + count operation.

    For + example:

    SELECT name,city,SUM(sales) FROM sales GROUP BY name,city +

    This query groups the information by the two fields ‘name’ and ‘city’ and produces a sum total of + these values. To translate this into a MapReduce function within Couchbase Server:

      +

    • From the list of selected fields, identify the field used for the calculation. These + will need to be exposed within the value emitted by the map() + function.

      • +

    • Identify the list of fields in the GROUP BY clause. These will need + to be output within the key of the map() function.

      • +

    • Identify the grouping function, for example SUM() or + COUNT(). You will need to use the equivalent built-in function, or a + custom function, within the reduce() function of the view.

      • +

    For example, in the above case, the corresponding map function can be written as + map() + :

    function(doc, meta) +{ + emit([doc.name,doc.city],doc.sales); +} +

    This + outputs the name and city as the key, and the sales as the value. Because the + SUM() function is used, the built-in reduce() function + _sum can be used.

    An example of this MapReduce combination can be seen _sum.

    More complex grouping operations may require a custom reduce function. +

    + + + +
    Translating SQL LIMIT and OFFSET

    Within SQL, the + LIMIT and OFFSET clauses to a given query are used as a + paging mechanism. For example, you might + use:

    SELECT recipeid,title FROM recipes LIMIT 100

    To + get the first 100 rows from the database, and then use the OFFSET to get + the subsequent groups of + records:

    SELECT recipeid,title FROM recipes LIMIT 100 OFFSET 100

    With + Couchbase Server, the limit and skip parameters when + supplied to the query provide the same basic + functionality:

    ?limit=100&skip=100

    Performance + for high values of skip can be affected.

    + + +     diff --git a/content/understanding-couchbase/views/views-writing-count.dita b/content/understanding-couchbase/views/views-writing-count.dita new file mode 100644 index 0000000000..069769413e --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-count.dita @@ -0,0 +1,56 @@ + + + + Built-in _count Function + The _count function provides a simple count of the input rows from the + map() function, using the keys and group level. + + +

    The keys and group level provides a count of correlated items + where the values generated during the map() stage are + ignored.

    +

    For example, using the input:

    + { + "rows" : [ + {"value" : 13000, "id" : "James", "key" : ["James", "Paris"] }, + {"value" : 20000, "id" : "James", "key" : ["James", "Tokyo"] }, + {"value" : 5000, "id" : "James", "key" : ["James", "Paris"] }, + {"value" : 7000, "id" : "Adam", "key" : ["Adam", "London"] }, + {"value" : 19000, "id" : "Adam", "key" : ["Adam", "Paris"] }, + {"value" : 17000, "id" : "Adam", "key" : ["Adam", "Tokyo"] }, + {"value" : 22000, "id" : "John", "key" : ["John", "Paris"] }, + {"value" : 3000, "id" : "John", "key" : ["John", "London"] }, + {"value" : 7000, "id" : "John", "key" : ["John", "London"] }, + ] +} + +

    Enabling the reduce() function and using a group level of 1 would + produce:

    + { + "rows" : [ + {"value" : 3, "key" : ["Adam" ] }, + {"value" : 3, "key" : ["James"] }, + {"value" : 3, "key" : ["John" ] } + ] +} + +

    The reduction has produce a new result set with the key as an array based on the first + element of the array from the map output. The value is the count of the number of + records collated by the first element.

    +

    For example, using a group level of 2 generates the following where the counts are for the + keys matching both the first two elements of the map output.:

    + { + "rows" : [ + {"value" : 1, "key" : ["Adam", "London"] }, + {"value" : 1, "key" : ["Adam", "Paris" ] }, + {"value" : 1, "key" : ["Adam", "Tokyo" ] }, + {"value" : 2, "key" : ["James","Paris" ] }, + {"value" : 1, "key" : ["James","Tokyo" ] }, + {"value" : 2, "key" : ["John", "London"] }, + {"value" : 1, "key" : ["John", "Paris" ] } + ] +} + +     +     + diff --git a/content/understanding-couchbase/views/views-writing-custom-reduce.dita b/content/understanding-couchbase/views/views-writing-custom-reduce.dita new file mode 100644 index 0000000000..a4dd4fe9de --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-custom-reduce.dita @@ -0,0 +1,63 @@ + + + + Custom Reduce Functions + The reduce() function has to work slightly differently to the + map() function. + + +

    In the primary form, a reduce() + function must convert the data supplied to it from the corresponding + map() function.

    +

    The core structure of the reduce function execution is shown the figure below.

    + + + + + +

    The base format of the reduce() function is as follows:

    + function(key, values, rereduce) { +… + +return retval; +} + +

    The reduce function is supplied three arguments:

    +
      +
    • key
      • +
    +

    The key is the unique key derived from the map() + function and the group_level parameter.

    +
      +
    • values
      • +
    +

    The values argument is an array of all of the values that match a + particular key. For example, if the same key is output three times, + data will be an array of three items containing, with each item + containing the value output by the emit() function.

    +
      +
    • rereduce
      • +
    +

    The rereduce indicates whether the function is being called as part of + a re-reduce, that is, the reduce function being called again to further reduce the input + data.

    +

    When rereduce is false:

    + * The supplied `key` argument will be an array where the first argument is the + `key` as emitted by the map function, and the `id` is the document ID that + generated the key. + +* The values is an array of values where each element of the array matches the + corresponding element within the array of `keys`. + +

    When rereduce is true:

    + * `key` will be null. + +* `values` will be an array of values as returned by a previous `reduce()` + function. + +

    The function should return the reduced version of the information by calling the + return() function. The format of the return value should match the + format required for the specified key.

    + +     +     diff --git a/content/understanding-couchbase/views/views-writing-map.dita b/content/understanding-couchbase/views/views-writing-map.dita new file mode 100644 index 0000000000..4c6e286a48 --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-map.dita @@ -0,0 +1,132 @@ + + + + Map Function + Map functions create a mapping between input data (JSON + objects) and data displayed in the view results (output). + + +

    Every document in the Couchbase bucket for the view is submitted to + the map() function in each view once, and it is the output from the + map() function that is used as the result of the view.

    +

    The map() function is supplied two arguments by the views processor. The + first argument is the JSON document data. The optional second argument is the associated + metadata for the document, such as the expiration, flags, and revision information.

    +

    The map function outputs zero or more ‘rows’ of information using an + emit() function. Each call to the emit() function is + equivalent to a row of data in the view result. The emit() function can be + called multiple times within the single pass of the map() function. This + functionality enables you to create views that may expose information stored in a compound + format within a single stored JSON record, for example generating a row for each item in an + array.

    + + +

    The map function is the most critical part of any view as it provides the logical + mapping between the input fields of the individual objects stored within Couchbase to + the information output when the view is accessed.

    +

    Through this mapping process, the map function and the view provide:

    +
      +
    • The output format and structure of the view on the bucket.
      • +
    • Structure and information used to query and select individual documents using the + view information.
      • +
    • Sorting of the view results.
      • +
    • Input information for summarizing and reducing the view content.
      • +
    + +

    Applications access views through the REST API, or through a Couchbase client library. + All client libraries provide a method for submitting a query into the view system and + obtaining and processing the results.

    + +

    The following diagram shows the name, salary and city fields of the stored + JSON documents translated into a table (an array of fields) in the generated view + content.

    + + + + + + +

    In this example, a map function is taking the Name, City, and Salary fields from the + JSON documents stored in the Couchbase bucket and mapping them to a table of these + fields. The map function which produces this output might look like this:

    + function(doc, meta) +{ + emit(doc.name, [doc.city, doc.salary]); +} + +

    When the view is generated the map() function is supplied two arguments + for each stored document, doc and meta :

    +
      +
    • doc +

      The stored document from the Couchbase bucket, either the JSON or binary content. + Content type can be identified by accessing the type field of the + meta argument object.

      • +
    • meta +

      The metadata for the stored document, containing expiry time, document ID, revision and + other information.

      +

      Every document in the Couchbase bucket is submitted to the map() + function in turn. After the view is created, only the documents created or changed since + the last update need to be processed by the view. View indexes and updates are + materialized when the view is accessed. Any documents added or changed since the last + access of the view will be submitted to the map() function again so + that the view is updated to reflect the current state of the data bucket.

      +

      Within the map() function itself you can perform any formatting, + calculation or other detail. To generate the view information, you use calls to the + emit() function. Each call to the emit() function + outputs a single row or record in the generated view content.

      +

      The emit() function accepts two arguments, the key and the value for + each record in the generated view:

      • +
    • key +

      The emitted key is used by Couchbase Server both for sorting and querying the content in + the database.

      +

      The key can be formatted in a variety of ways, including as a string or compound value + (such as an array or JSON object). The content and structure of the key is important, + because it is through the emitted key structure that information is selected within the + view.

      +

      All views are output in a sorted order according to the content and structure of the + key. Keys using a numeric value are sorted numerically, for strings, UTF-8 is used. Keys + can also support compound values such as arrays and hashes.

      +

      The key content is used for querying by using a combination of this sorting process and + the specification of either an explicit key or key range within the query specification. + For example, if a view outputs the RECIPE TITLE field as a key, you + could obtain all the records matching ‘Lasagne’ by specifying that only the keys + matching ‘Lasagne’ are returned.

      • +
    • value +

      The value is the information that you want to output in each view row. The value can be + anything, including both static data, fields from your JSON objects, and calculated + values or strings based on the content of your JSON objects.

      • +
    + +

    The content of the value is important when performing a reduction, since it is the value + that is used during reduction, particularly with the built-in reduction functions. For + example, when outputting sales data, you might put the SALESMAN into + the emitted key, and put the sales amounts into the value. The built-in + _sum function will then total up the content of the corresponding + value for each unique key.

    +

    The format of both key and value is up to you. You can format these as single values, + strings, or compound values such as arrays or JSON. The structure of the key is + important because you must specify keys in the same format as they were generated in the + view specification.

    +

    The emit() function can be called multiple times in a single map + function, with each call outputting a single row in the generated view. This can be + useful when you want to supporting querying information in the database based on a + compound field. For a sample view definition and selection criteria.

    +

    Views and map generation are also very forgiving. If you elect to output fields in from + the source JSON objects that do not exist, they are replaced with a + null value, rather than generating an error.

    +

    For example, in the view below, some of the source records do contain all of the fields + in the specified view. The result in the view result is just the null + entry for that field in the value output.

    + + + + + +

    You should check that the field or data source exists during the map processing before + emitting the data.

    +

    To better understand how the map function works to output different types of information + and retrieve it, see View and Query Pattern Samples.

    + + +     +     diff --git a/content/understanding-couchbase/views/views-writing-nonjson.dita b/content/understanding-couchbase/views/views-writing-nonjson.dita new file mode 100644 index 0000000000..31fa1be05d --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-nonjson.dita @@ -0,0 +1,43 @@ + + + + Views for Non-JSON Objects + Views on non_JSON data enable range queries on the key data. + + +

    If the data stored within your buckets is not JSON formatted or JSON in nature, then the + information is stored in the database as an attachment to a JSON document returned by + the core database layer.

    +

    This does not mean that you cannot create views on the information, but it does limit + the information that you can output with your view to the information exposed by the + document key used to store the information.

    +

    At the most basic level, this means that you can still do range queries on the key + information. For example:

    + function(doc, meta) +{ + emit(meta.id, null); +} + +

    You can now perform range queries by using the emitted key data and an appropriate + startkey and endkey value.

    +

    If you use a structured format for your keys, for example using a prefix for the data + type, or separators used to identify different elements, then your view function can + output this information explicitly in the view. For example, if you use a key structure + where the document ID is defined as a series of values that are colon separated:

    + OBJECTYPE:APPNAME:OBJECTID + +

    You can parse this information within the JavaScript map/reduce query to output each + item individually. For example:

    + function(doc, meta) +{ + values = meta.id.split(':',3); + emit([values[0], values[1], values[2]], null); +} + +

    The above function will output a view that consists of a key containing the object type, + application name, and unique object ID. You can query the view to obtain all entries of + a specific object type using:

    + startkey=['monster', null, null]&endkey=['monster','\u0000' ,'\u0000'] + +     +     diff --git a/content/understanding-couchbase/views/views-writing-reduce.dita b/content/understanding-couchbase/views/views-writing-reduce.dita new file mode 100644 index 0000000000..637280e923 --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-reduce.dita @@ -0,0 +1,147 @@ + + + + Reduce Function + Reduce functions are used to summarize the content generated during the map phase. + + +

    Reduce + functions are optional in a view and do not have to be defined. When they exist, each row + of output (from each emit() call in the corresponding + map() function) is processed by the corresponding + reduce() function.

    +

    If a reduce function is specified in the view definition it is automatically used. You can + access a view without enabling the reduce function by disabling reduction ( + reduce=false ) when the view is accessed.

    +

    Typical uses for a reduce function are to produce a summarized count of the input data, or + to provide sum or other calculations on the input data. For example, if the input data + included employee and salary data, the reduce function could be used to produce a count of + the people in a specific location, or the total of all the salaries for people in those + locations.

    +

    The combination of the map and the reduce function produce the corresponding view. The two + functions work together, with the map producing the initial material based on the content + of each JSON document, and the reduce function summarizing the information generated during + the map phase. The reduction process is selectable at the point of accessing the view, you + can choose whether to the reduce the content or not, and, by using an array as the key, you + can specifying the grouping of the reduce information.

    +

    Each row in the output of a view consists of the view key and the view value. When + accessing a view using only the map function, the contents of the view key and value are + those explicitly stated in the definition. In this mode the view will also always contain + an id field which contains the document ID of the source record (i.e. the + string used as the ID when storing the original data record).

    +

    When accessing a view employing both the map and reduce functions the key and value are + derived from the output of the reduce function based on the input key and group level + specified. A document ID is not automatically included because the document ID cannot be + determined from reduced data where multiple records may have been merged into one.

    + +

    The following diagram shows an example of the view creation process:

    + + + + + +

    Because of the separation of the two elements, the two functions can be considered + individually.

    +

    For information on how to write map functions, and how the output of the map function + affects and supports searching.

    +

    View names must be specified using one or more UTF-8 characters. You cannot have a blank + view name. View names cannot have leading or trailing whitespace characters (space, tab, + newline, or carriage-return).

    +

    To create views, you can use either the Web Console View editor, use the REST API for design + documents, or use one of the client libraries that support view management.

    + +

    Often the information that you are searching or reporting on needs to be summarized or + reduced. There are a number of different occasions when this can be useful. For example, + if you want to obtain a count of all the items of a particular type, such as comments, + recipes matching an ingredient, or blog entries against a keyword.

    +

    When using a reduce function in your view, the value that you specify in the call to + emit() is replaced with the value generated by the reduce function. + This is because the value specified by emit() is used as one of the + input parameters to the reduce function. The reduce function is designed to reduce a + group of values emitted by the corresponding map() function.

    +

    Alternatively, reduce can be used for performing sums, for example adding up all the invoice + values for a single client or adding up the preparation and cooking times in a recipe. Any + calculation that can be performed on a group of the emitted data.

    +

    In each of the above cases, the raw data is the information from one or more rows of + information produced by a call to emit(). The input data, each record + generated by the emit() call, is reduced and grouped together to + produce a new record in the output.

    +

    The grouping is performed based on the value of the emitted key, with the rows of + information generated during the map phase being reduced and collated according to the + uniqueness of the emitted key.

    +

    When using a reduce function the reduction is applied as follows:

    +
      +
    • For each record of input, the corresponding reduce function is applied on the row, + and the return value from the reduce function is the resulting row.
      • +
    +

    For example, using the built-in _sum reduce function, the + value in each case would be totaled based on the emitted key:

    + +{ + "rows" : [ + {"value" : 13000, "id" : "James", "key" : "James" }, + {"value" : 20000, "id" : "James", "key" : "James" }, + {"value" : 5000, "id" : "Adam", "key" : "Adam" }, + {"value" : 8000, "id" : "Adam", "key" : "Adam" }, + {"value" : 10000, "id" : "John", "key" : "John" }, + {"value" : 34000, "id" : "John", "key" : "John" } + ] +} + + +

    Using the unique key of the name, the data generated by the map above would be reduced, + using the key as the collator, to the produce the following output:

    + +{ + "rows" : [ + {"value" : 33000, "key" : "James" }, + {"value" : 13000, "key" : "Adam" }, + {"value" : 44000, "key" : "John" }, + ] +} + +

    In each case the values for the common keys (John, Adam, James), have been totalled, and + the six input rows reduced to the 3 rows shown here.

    +
      +
    • +

      Results are grouped on the key from the call to emit() if + grouping is selected during query time. As shown in the previous example, the + reduction operates by the taking the key as the group value as using this as the + basis of the reduction.

      +
      • +
    • +

      If you use an array as the key, and have selected the output to be grouped during + querying you can specify the level of the reduction function, which is analogous + to the element of the array on which the data should be grouped.

      +
      • +
    +

    The view definition is flexible. You can select whether the reduce function is applied + when the view is accessed. This means that you can access both the reduced and unreduced + (map-only) content of the same view. You do not need to create different views to access + the two different types of data.

    +

    Whenever the reduce function is called, the generated view content contains the same key + and value fields for each row, but the key is the selected group (or an array of the + group elements according to the group level), and the value is the computed reduction + value.

    +

    Couchbase includes the following built-in reduce functions:

    +
      +
    • _count
      • +
    • _sum
      • +
    • _stats.
      • +
    + You can also write your own custom reduction functions. +

    The reduce function also has a final additional benefit. The results of the computed + reduction are stored in the index along with the rest of the view information. This + means that when accessing a view with the reduce function enabled, the information comes + directly from the index content. This results in a very low impact on the Couchbase + Server to the query (the value is not computed at runtime), and results in very fast + query times, even when accessing information based on a range-based query.

    +

    The reduce() function is designed to reduce and summarize the data + emitted during the map() phase of the process. It should only be used + to summarize the data, and not to transform the output information or concatenate the + information into a single structure.

    +

    When using a composite structure, the size limit on the composite structure within the + reduce() function is 64KB.

    + +     +     diff --git a/content/understanding-couchbase/views/views-writing-rereduce.dita b/content/understanding-couchbase/views/views-writing-rereduce.dita new file mode 100644 index 0000000000..b7f0030844 --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-rereduce.dita @@ -0,0 +1,96 @@ + + + + Re-reduce Argument + + + +

    For reduce() functions, they should be both transparent and standalone. + For example, the _sum function did not rely on global variables or + parsing of existing data, and didn’t need to call itself, hence it is also + transparent.

    +

    In order to handle incremental map/reduce functionality (i.e. updating an existing + view), each function must also be able to handle and consume the functions own output. + This is because in an incremental situation, the function must be handle both the new + records, and previously computed reductions.

    +

    This can be explicitly written as follows:

    + f(keys, values) = f(keys, [ f(keys, values) ]) +

    The following diagram shows previous reductions included within the array of information that + are re-supplied to the reduce function as an element of the array of values supplied to the + reduce function.

    + + + +

    That is, the input of a reduce function can be not only the raw data from the map phase, + but also the output of a previous reduce phase. This is called + rereduce, and can be identified by the third argument to the + reduce(). When the rereduce argument is true, both + the key and values arguments are arrays, with the + corresponding element in each containing the relevant key and value. I.e., + key[1] is the key related to the value of + value[1].

    +

    An example of this can be seen by considering an expanded version of the + sum function showing the supplied values for the first iteration of + the view index building:

    + function('James', [ 13000,20000,5000 ]) {...} + +

    When a document with the ‘James’ key is added to the database, and the view operation is + called again to perform an incremental update, the equivalent call is:

    + function('James', [ 19000, function('James', [ 13000,20000,5000 ]) ]) { ... } + +

    In reality, the incremental call is supplied the previously computed value, and the + newly emitted value from the new document:

    + function('James', [ 19000, 38000 ]) { ... } + +

    Fortunately, the simplicity of the structure for sum means that the + function both expects an array of numbers, and returns a number, so these can easily be + recombined.

    +

    If writing more complex reductions, where a compound key is output, the + reduce() function must be able to handle processing an argument of + the previous reduction as the compound value in addition to the data generated by the + map() phase. For example, to generate a compound output showing both + the total and count of values, a suitable reduce() function could be + written like this:

    + function(key, values, rereduce) { + var result = {total: 0, count: 0}; + for(i=0; i < values.length; i++) { + if(rereduce) { + result.total = result.total + values[i].total; + result.count = result.count + values[i].count; + } else { + result.total = sum(values); + result.count = values.length; + } + } + return(result); +} + +

    Each element of the array supplied to the function is checked using the built-in + typeof function to identify whether the element was an object (as + output by a previous reduce), or a number (from the map phase), and then updates the + return value accordingly.

    +

    Using the sample sales data, and group level of two, the output from a reduced view may + look like this:

    + {"rows":[ +{"key":["Adam", "London"],"value":{"total":7000, "count":1}}, +{"key":["Adam", "Paris"], "value":{"total":19000, "count":1}}, +{"key":["Adam", "Tokyo"], "value":{"total":17000, "count":1}}, +{"key":["James","Paris"], "value":{"total":118000,"count":3}}, +{"key":["James","Tokyo"], "value":{"total":20000, "count":1}}, +{"key":["John", "London"],"value":{"total":10000, "count":2}}, +{"key":["John", "Paris"], "value":{"total":22000, "count":1}} +] +} + +

    Reduce functions must be written to cope with this scenario in order to cope with the + incremental nature of the view and index building. If this is not handled correctly, the + index will fail to be built correctly.

    +

    The reduce() function is designed to reduce and summarize the data + emitted during the map() phase of the process. It should only be used + to summarize the data, and not to transform the output information or concatenate the + information into a single structure.

    +

    When using a composite structure, the size limit on the composite structure within the + reduce() function is 64KB.

    + +     +     diff --git a/content/understanding-couchbase/views/views-writing-rewriting.dita b/content/understanding-couchbase/views/views-writing-rewriting.dita new file mode 100644 index 0000000000..53f439b421 --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-rewriting.dita @@ -0,0 +1,49 @@ + + + + Re-writing Built-in Reduce Functions + + + +

    Using this model as a template, it is possible to write the full implementation of the + built-in functions _sum and _count when working with + the sales data and the standard map() function below:

    + function(doc, meta) +{ + emit(meta.id, null); +} + +

    The _count function returns a count of all the records for a given key. + Since argument for the reduce function contains an array of all the values for a given + key, the length of the array needs to be returned in the reduce() + function:

    + function(key, values, rereduce) { + if (rereduce) { + var result = 0; + for (var i = 0; i < values.length; i++) { + result += values[i]; + } + return result; + } else { + return values.length; + } +} + +

    To explicitly write the equivalent of the built-in _sum reduce + function, the sum of supplied array of values needs to be returned:

    + function(key, values, rereduce) { + var sum = 0; + for(i=0; i < values.length; i++) { + sum = sum + values[i]; + } + return(sum); +} + +

    In the above function, the array of data values is iterated over and added up, with the + final value being returned.

    + + + + +     +     diff --git a/content/understanding-couchbase/views/views-writing-stats.dita b/content/understanding-couchbase/views/views-writing-stats.dita new file mode 100644 index 0000000000..d05e4dadbc --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-stats.dita @@ -0,0 +1,60 @@ + + + + Built-in _stats Function + The built-in _stats reduce function produces statistical calculations + for the input data. + + +

    As with the _sum function, the corresponding value + in the emit call should be a number. The generated statistics include the sum, count, + minimum ( min ), maximum ( max ) and sum squared ( + sumsqr ) of the input rows.

    + +

    For example, using the sales data, a slightly truncated output at group level one shows + the same fields in the output value for each of the reduced output rows.

    + { + "rows" : [ + { + "value" : { + "count" : 3, + "min" : 7000, + "sumsqr" : 699000000, + "max" : 19000, + "sum" : 43000 + }, + "key" : [ + "Adam" + ] + }, + { + "value" : { + "count" : 3, + "min" : 5000, + "sumsqr" : 594000000, + "max" : 20000, + "sum" : 38000 + }, + "key" : [ + "James" + ] + }, + { + "value" : { + "count" : 3, + "min" : 3000, + "sumsqr" : 542000000, + "max" : 22000, + "sum" : 32000 + }, + "key" : [ + "John" + ] + } + ] +} + + + +     +     diff --git a/content/understanding-couchbase/views/views-writing-sum.dita b/content/understanding-couchbase/views/views-writing-sum.dita new file mode 100644 index 0000000000..e13ec878e6 --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-sum.dita @@ -0,0 +1,44 @@ + + + + Built-in _sum Function + The built-in _sum function sums the values from the + map() function call by summing up the information in the + value for each row. + + +

    The information in the value for each row can be either a single number or during a re-reduce, + an array of numbers.

    +

    The input values must be a number, not a string-representation of a number. The entire + map/reduce will fail if the reduce input is not in the correct format. You should use + the parseInt() or parseFloat() function calls within + your map() function stage to ensure that the input data is a + number.

    +

    For example, using the same sales source data, accessing the group level 1 view would + produce the total sales for each salesman:

    + { + "rows" : [ + {"value" : 43000, "key" : [ "Adam" ] }, + {"value" : 38000, "key" : [ "James" ] }, + {"value" : 32000, "key" : [ "John" ] } + ] +} + +

    Using a group level of 2 you get the information summarized by salesman and city:

    + { + "rows" : [ + {"value" : 7000, "key" : [ "Adam", "London" ] }, + {"value" : 19000, "key" : [ "Adam", "Paris" ] }, + {"value" : 17000, "key" : [ "Adam", "Tokyo" ] }, + {"value" : 18000, "key" : [ "James", "Paris" ] }, + {"value" : 20000, "key" : [ "James", "Tokyo" ] }, + {"value" : 10000, "key" : [ "John", "London" ] }, + {"value" : 22000, "key" : [ "John", "Paris" ] } + ] +} + + + +     +     + diff --git a/content/understanding-couchbase/views/views-writing-utility.dita b/content/understanding-couchbase/views/views-writing-utility.dita new file mode 100644 index 0000000000..01d9c94d13 --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-utility.dita @@ -0,0 +1,41 @@ + + + + Built-in Utility Functions + Utility functions are available that can be used within map() and + reduce() functions. + + +

    Couchbase Server incorporates different utility functions beyond the core JavaScript + functionality that can be used within map() and + reduce() functions where relevant.

    +
      +
    • dateToArray(date)
      • +
    +

    Converts a JavaScript Date object or a valid date string such as + "2012-07-30T23:58:22.193Z" into an array of individual date components. For example, the + previous string would be converted into a JavaScript array:

    + +[2012, 7, 30, 23, 58, 22] + +

    The function can be particularly useful when building views using dates as the key where + the use of a reduce function is being used for counting or rollup.

    +

    Currently, the function works only on UTC values. Timezones are not supported.

    +
      +
    • decodeBase64(doc)
      • +
    +

    Converts a binary (base64) encoded value stored in the database into a string. This can + be useful if you want to output or parse the contents of a document that has not been + identified as a valid JSON value.

    +
      +
    • sum(array)
      • +
    +

    When supplied with an array containing numerical values, each value is summed and the + resulting total is returned.

    +

    For example:

    + +sum([12,34,56,78]) + + +     +     diff --git a/content/understanding-couchbase/views/views-writing-views.dita b/content/understanding-couchbase/views/views-writing-views.dita new file mode 100644 index 0000000000..de4138b7e3 --- /dev/null +++ b/content/understanding-couchbase/views/views-writing-views.dita @@ -0,0 +1,287 @@ + + + + Views Best Practices + Several practices should be kept in mind with developing and deploying + views. + + +

    Although you are free to write views matching your data, you should keep in mind the + performance and storage implications of creating and organizing the different design + document and view definitions.

    +

    Keep in mind the following factors when developing and deploying views:

    +
      +
    • View quality per design document
      • +
    • Modifying existing views
      • +
    • Don’t include document IDs
      • +
    • Check document fields
      • +
    • View Size, disk storage and I/O
      • +
    • Include value data in views
      • +
    • Don’t include entire documents in view output
      • +
    • Use document types
      • +
    • Use built-in Reduce functions
      • +
    + +
    View quality per design document + +

    Because the index for each map/reduce combination within each view within a given design + document is updated at the same time, avoid declaring too many views within the same + design document. For example, if you have a design document with five different views, + all five views will be updated simultaneously, even if only one of the views is + accessed.

    +

    This can result in increase view index generation times, especially for frequently + accessed views. Instead, move frequently used views out to a separate design + document.

    +

    The exact number of views per design document should be determined from a combination of + the update frequency requirements on the included views and grouping of the view + definitions. For example, if you have a view that needs to be updated with a high + frequency (for example, comments on a blog post), and another view that needs to be + updated less frequently (e.g. top blogposts), separate the views into two design + documents so that the comments view can be updated frequently, and independently, of the + other view.

    +

    You can always configure the updating of the view through the use of the + stale parameter. You can also configure different automated view + update times for individual design documents

    + +
    + + +
    Modifying existing views + +

    If you modify an existing view definition, or are executing a full build on a + development view, the entire view will need to be recreated. In addition, all the views + defined within the same design document will also be recreated.

    +

    Rebuilding all the views within a single design document is an expensive operation in + terms of I/O and CPU requirements, as each document will need to be parsed by each views + map() and reduce() functions, with the resulting + index stored on disk.

    +

    This process of rebuilding will occur across all the nodes within the cluster and + increases the overall disk I/O and CPU requirements until the view has been recreated. + This process will take place in addition to any production design documents and views + that also need to be kept up to date.

    + +
    + + + +
    Don’t include document IDs + +

    The document ID is automatically output by the view system when the view is accessed. + When accessing a view without reduce enabled you can always determine the document ID of + the document that generated the row. You should not include the document ID (from + meta.id ) in your key or value data.

    +
    + + +
    Check document fields + +

    Fields and attributes from source documentation in map() or + reduce() functions should be checked before their value is checked + or compared. This can cause issues because the view definitions in a design document are + processed at the same time. A common cause of runtime errors in views is missing or + invalid field and attribute checking.

    +

    The most common issue is a field within a null object being accessed. This generates a + runtime error that will cause execution of all views within the design document to fail. + To address this problem, you should check for the existence of a given object before it + is used, or the content value is checked. For example, the following view will fail if + the doc.ingredient object does not exist, because accessing the + length attribute on a null object will fail:

    + + +function(doc, meta) +{ + emit(doc.ingredient.ingredtext, null); +} + + +

    Adding a check for the parent object before calling emit() ensures that + the function is not called unless the field in the source document exists:

    + +function(doc, meta) +{ + if (doc.ingredient) + { + emit(doc.ingredient.ingredtext, null); + } +} + +

    The same check should be performed when comparing values within the if + statement.

    +

    This test should be performed on all objects where you are checking the attributes or + child values (for example, indices of an array).

    +
    + +
    View Size, disk storage and I/O + +

    Within the map function, the information declared within your emit() + statement is included in the view index data and stored on disk. Outputting this information has + the following effects on your indexes:

    + * *Increased index size on disk* — More detailed or complex key/value combinations + in generated views will result in more information being stored on disk. + +* *Increased disk I/O* — in order to process and store the information on disk, + and retrieve the data when the view is queried. A larger more complex key/value + definition in your view will increase the overall disk I/O required both to + update and read the data back. + + +

    The result is that the index can be quite large, and in some cases, the size of the + index can exceed the size of the original source data by a significant factor if + multiple views are created, or you include large portions or the entire document data in + the view output.

    +

    For example, if each view contains the entire document as part of the value, and you + define ten views, the size of your index files will be more than 10 times the size of + the original data on which the view was created. With a 500-byte document and 1 million + documents, the view index would be approximately 5GB with only 500MB of source data.

    +
    +
    Limits on document sizes for indexing + +

    These are limits on the document size during indexing:

    + + + + + + + Limits for indexing + + + + + + + Parameter + Default value + Description + + + + + indexer_max_doc_size + 20M + The view engine enforced a limit of 1 MB on documents that can be indexed in Couchbase + Server version 2.x. In version 3.x, the limit is was increased to 20 MB to ensure every + document gets indexed and not silently dropped by the view engine if size of document + exceeded previously enforced limit. + + + max_kv_size_per_doc + 1M + The maximum byte size allowed to be emitted for a single document and per view. This + is the sum of the sizes of all emitted keys and values. If a document emits a key, if the + value pair exceeds max_kv_size_per_doc an error is logged and that document + is not indexed. A value of 0 for this new setting disables the limit + (meaning unlimited, as it was before this change). + + + function_timeout + 1000ms + Maximum duration, in milliseconds, for the execution time of all the map/reduce + functions in a design document against a single document (map function), or against a list + of map values/reductions (reduce/rereduce function). If map/map+reduce exceeds + function_timeout it is aborted and this document is not indexed. + + + +
    + + + +
    + +
    Include value data in views + +

    Views store both the key and value emitted by the emit(). To ensure the + highest performance, views should only emit the minimum key data required to search and + select information. The value output by emit() should only be used when + you need the data to be used within a reduce().

    +

    You can obtain the document value by using the core Couchbase API to get individual + documents or documents in bulk. Some SDKs can perform this operation for you + automatically.

    +

    Using this model will also prevent issues where the emitted view data may be + inconsistent with the document state and your view is emitting value data from the + document which is no longer stored in the document itself.

    +

    For views that are not going to be used with reduce, you should output a null value:

    + +function(doc, meta) + { + if(doc.type == 'object') + emit(doc.experience, null); + } + + +

    This creates an optimized view containing only the information required, ensuring + the highest performance when updating the view, and smaller disk usage.

    +
    + +
    Don’t include entire documents in view output + +

    A view index should be designed to provide base information and through the implicitly + returned document ID point to the source document. It is bad practice to include the + entire document within your view output.

    +

    You can always access the full document data through the client libraries by later + requesting the individual document data. This is typically much faster than including + the full document data in the view index, and enables you to optimize the index + performance without sacrificing the ability to load the full document data.

    +

    For example, the following is an example of a bad view:

    + +function(doc, meta) + { + if(doc.type == 'object') + emit(doc.experience, doc); + } + +

    The above view may have significant performance and index size effects.

    +

    This will include the full document content in the index.

    +

    Instead, the view should be defined as:

    + +function(doc, meta) + { + if(doc.type == 'object') + emit(doc.experience, null); + } + +

    You can then either access the document data individually through the client libraries, + or by using the built-in client library option to separately obtain the document + data.

    +
    + + +
    Use document types + +

    If you are using a document type (by using a field in the stored JSON to indicate the + document structure), be aware that on a large database this can mean that the view + function is called to update the index for document types that are not being updated or + added to the index.

    +

    For example, within a database storing game objects with a standard list of objects, and + the users that interact with them, you might use a field in the JSON to indicate + ‘object’ or ‘player’. With a view that outputs information when the document is an + object:

    + +function(doc, meta) +{ + emit(doc.experience, null); +} + +

    If only players are added to the bucket, the map/reduce functions to update this view + will be executed when the view is updated, even though no new objects are being added to + the database. Over time, this can add a significant overhead to the view building + process.

    +

    In a database organization like this, it can be easier from an application perspective + to use separate buckets for the objects and players, and therefore completely separate + view index update and structure without requiring to check the document type during + progressing.

    +
    + + + +
    Use built-in Reduce functions + +

    Where possible, use one of the supplied built-in reduce functions, + _sum, _count](#couchbase-views-writing-reduce-count), + _stats](#couchbase-views-writing-reduce-stats).

    +

    These functions are highly optimized. Using a custom reduce function requires additional + processing and may impose additional build time on the production of the index.

    +
    +     +     diff --git a/content/understanding-couchbase/views/views-writing.dita b/content/understanding-couchbase/views/views-writing.dita new file mode 100644 index 0000000000..24783aec89 --- /dev/null +++ b/content/understanding-couchbase/views/views-writing.dita @@ -0,0 +1,34 @@ + + + + Writing MapReduce Views + During the view creation process, the output structure, field order, content, and any + summary or grouping information desired in the view is defined. + +

    The fundamentals of a view are straightforward. A view creates a perspective on the data + stored in Couchbase buckets in a format that can be used to represent the data in a + specific way, define and filter the information, and provide a basis for searching or + querying the data in the database based on the content.

    +

    Views achieve this by defining an output structure that translates the stored JSON object + data into a JSON array or object across two components, the key and the value. This + definition is performed through the specification of two separate functions written in + JavaScript. The view definition is divided into two parts, a map function and a reduce + function: + +

      +
    • Map functions
      • +
    • Reduce functions
      • +

    + + + + + + + + Couchbase SDKs + + + +     diff --git a/content/understanding-couchbase/views/views.ditamap b/content/understanding-couchbase/views/views.ditamap new file mode 100644 index 0000000000..520ba5df75 --- /dev/null +++ b/content/understanding-couchbase/views/views.ditamap @@ -0,0 +1,40 @@ + + + + Views + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/views/views-basics.dita b/content/views/views-basics.dita index 5ce7acc806..529bc88b7f 100644 --- a/content/views/views-basics.dita +++ b/content/views/views-basics.dita @@ -26,7 +26,7 @@ costly schema update before doing so.

    In the following example, the view takes the Name, City and Salary fields from the documents - and creates a array of this information for each document in the view. A view is created by + and creates an array of this information for each document in the view. A view is created by iterating over every single document within the Couchbase bucket and outputting the specified information. The resulting view file is stored for future use and updated with new data when the view is accessed. The process is incremental and therefore has a low diff --git a/content/xdcr/picts/XdcrCbRecoveryLocalServersBackUpAgain.png b/content/xdcr/picts/XdcrCbRecoveryLocalServersBackUpAgain.png new file mode 100644 index 0000000000..651b6f45d0 Binary files /dev/null and b/content/xdcr/picts/XdcrCbRecoveryLocalServersBackUpAgain.png differ diff --git a/content/xdcr/picts/XdcrCbRecoveryLocalServersDown.png b/content/xdcr/picts/XdcrCbRecoveryLocalServersDown.png new file mode 100644 index 0000000000..11edcc8688 Binary files /dev/null and b/content/xdcr/picts/XdcrCbRecoveryLocalServersDown.png differ diff --git a/content/xdcr/picts/XdcrCbRecoveryLocalServersUp.png b/content/xdcr/picts/XdcrCbRecoveryLocalServersUp.png new file mode 100644 index 0000000000..dac35c633d Binary files /dev/null and b/content/xdcr/picts/XdcrCbRecoveryLocalServersUp.png differ diff --git a/content/xdcr/picts/XdcrCbRecoveryReplicationScreen.png b/content/xdcr/picts/XdcrCbRecoveryReplicationScreen.png new file mode 100644 index 0000000000..01bfae20fe Binary files /dev/null and b/content/xdcr/picts/XdcrCbRecoveryReplicationScreen.png differ diff --git a/content/xdcr/picts/addRemoteClusterDialog.png b/content/xdcr/picts/addRemoteClusterDialog.png new file mode 100644 index 0000000000..742bb6c137 Binary files /dev/null and b/content/xdcr/picts/addRemoteClusterDialog.png differ diff --git a/content/xdcr/picts/editRemoteClusterDialog.png b/content/xdcr/picts/editRemoteClusterDialog.png new file mode 100644 index 0000000000..a6cea5d6e6 Binary files /dev/null and b/content/xdcr/picts/editRemoteClusterDialog.png differ diff --git a/content/xdcr/picts/editRemoteClusterDialogWithFullConnectionByCerts.png b/content/xdcr/picts/editRemoteClusterDialogWithFullConnectionByCerts.png new file mode 100644 index 0000000000..19173b0df2 Binary files /dev/null and b/content/xdcr/picts/editRemoteClusterDialogWithFullConnectionByCerts.png differ diff --git a/content/xdcr/picts/editRemoteClusterDialogWithFullConnectionByPwd.png b/content/xdcr/picts/editRemoteClusterDialogWithFullConnectionByPwd.png new file mode 100644 index 0000000000..ed74c494b0 Binary files /dev/null and b/content/xdcr/picts/editRemoteClusterDialogWithFullConnectionByPwd.png differ diff --git a/content/xdcr/picts/xccr-create.png b/content/xdcr/picts/xccr-create.png index e7eccbbce8..5e32e4630b 100644 Binary files a/content/xdcr/picts/xccr-create.png and b/content/xdcr/picts/xccr-create.png differ diff --git a/content/xdcr/picts/xdcr-create-replication.png b/content/xdcr/picts/xdcr-create-replication.png index d91b6a754d..a53811f09d 100644 Binary files a/content/xdcr/picts/xdcr-create-replication.png and b/content/xdcr/picts/xdcr-create-replication.png differ diff --git a/content/xdcr/picts/xdcr-edit-remote-cluster.png b/content/xdcr/picts/xdcr-edit-remote-cluster.png new file mode 100644 index 0000000000..1e555173f7 Binary files /dev/null and b/content/xdcr/picts/xdcr-edit-remote-cluster.png differ diff --git a/content/xdcr/xdcr-adv-settings.dita b/content/xdcr/xdcr-adv-settings.dita index 8ca1d871a4..acba7faf9a 100644 --- a/content/xdcr/xdcr-adv-settings.dita +++ b/content/xdcr/xdcr-adv-settings.dita @@ -28,7 +28,7 @@ less than or equal to the number of XDCR Target Nozzles per Node.

    A small value of two or four is often sufficient. The default is two and the value range is 1-100. The CLI command for setting this value is provided in .
    + href="../cli/cbcli/couchbase-cli-xdcr-replicate.html" format="html"/>. diff --git a/content/xdcr/xdcr-change-encryption.dita b/content/xdcr/xdcr-change-encryption.dita index 84e2179754..6232810c67 100644 --- a/content/xdcr/xdcr-change-encryption.dita +++ b/content/xdcr/xdcr-change-encryption.dita @@ -25,6 +25,6 @@ - + diff --git a/content/xdcr/xdcr-conflict-resolution.dita b/content/xdcr/xdcr-conflict-resolution.dita index 62902ddec3..5e2bb7e900 100644 --- a/content/xdcr/xdcr-conflict-resolution.dita +++ b/content/xdcr/xdcr-conflict-resolution.dita @@ -23,8 +23,10 @@

    +
    How Conflict Resolution Works +

    When a document is modified at an XDCR source, it determines if this revision of the document should be applied to the destination. XDCR fetches metadata for the document from the destination cluster before it replicates the document. XDCR compares the destination @@ -43,7 +45,10 @@

    Conflict resolution is an automatic process and does not require any manual correction or selection of documents.

    -
    Revision-based Conflict Resolution + +
    + Revision-based Conflict Resolution +

    Revision ID-based conflict resolution uses revision ID as the first field to resolve conflicts between two writes across clusters. Revision IDs are maintained per document and are incremented with every update to the document. The revision ID monotonically keeps track @@ -66,8 +71,11 @@ recent mutation may not be necessarily the one that wins conflict resolution using revision ID.

    +
    + Timestamp-based Conflict Resolution +

    Timestamp-based conflict resolution uses the document timestamp (stored in the CAS) as the first field to resolve conflicts between two writes across clusters. Couchbase Server uses a hybrid logical clock (HLC), a combination of a physical clock and a logical clock, to store @@ -99,8 +107,13 @@

    See for further details about using timestamp-based conflict resolution including important environmental considerations.

    -
    Choosing the Right Conflict Resolution - Method

    The conflict resolution policy is configured on a per-bucket basis at + +

    + Choosing the Right Conflict Resolution + Method + +

    + The conflict resolution policy is configured on a per-bucket basis at bucket creation time, it cannot be changed later. For more information, see . It is therefore important to choose the correct conflict resolution method for your application @@ -119,7 +132,9 @@ writing to the same key repeatedly. In the event of a conflict occurring, you would want to keep the version of the document which was captured most recently, as that is the most accurate 'current' temperature. Timestamp-based conflict resolution ensures that the most - recent version of the document would be used.

    + recent version of the document would be used.

    + +

    It is not possible to create XDCR replications between buckets with different conflict resolution policies.

    @@ -130,6 +145,10 @@ should be used in all cases where you need to use XDCR from pre-4.6 Couchbase Server clusters.

    + +
    + + diff --git a/content/xdcr/xdcr-create-reference.dita b/content/xdcr/xdcr-create-reference.dita index 98a20ecfd9..47fa74b6f2 100644 --- a/content/xdcr/xdcr-create-reference.dita +++ b/content/xdcr/xdcr-create-reference.dita @@ -46,7 +46,7 @@ - + diff --git a/content/xdcr/xdcr-create-replication.dita b/content/xdcr/xdcr-create-replication.dita index ec5ede1a5f..0c1e4cd016 100644 --- a/content/xdcr/xdcr-create-replication.dita +++ b/content/xdcr/xdcr-create-replication.dita @@ -17,14 +17,13 @@
  • In the section To, select a destination cluster and enter the bucket name from the destination cluster:
  • Select the check box Enable Advanced filtering. This will allow you - to specify the filtering expression while creating replication.

    For more details, see - .

    • + to specify the filtering expression while creating replication.
  • Configure XDCR advanced settings.
  • Click the Replicate button to start the replication process.
    • - + diff --git a/content/xdcr/xdcr-create.dita b/content/xdcr/xdcr-create.dita index 4d3a9ed080..eed280da01 100644 --- a/content/xdcr/xdcr-create.dita +++ b/content/xdcr/xdcr-create.dita @@ -1,13 +1,25 @@ - Managing XDCRXDCR replications are configured on a per-bucket basis. + + + Managing XDCR + + + + Cross Data-Center Replication can be managed by means of the Couchbase Web Console, the + Couchbase CLI, or the Couchbase REST API. + + -

    Full, Cluster, and Replication Administrators can create, edit, and delete cluster references - and replications.

    -

    From the Couchbase Web Console > XDCR menu, configure the - replications. As replication is on a per bucket basis, you must configure replication for each - bucket to each target cluster.

    + +

    + If you possess the role of Full, Cluster, or XDCR Administrator, you can create, edit, and delete cluster references + and replications from the Couchbase Web Console > XDCR menu. As replication + is on a per bucket basis, you must configure replication for each + bucket individually, to each target cluster. +

    +

    Before configuring XDCR:

    • Configure all nodes within each cluster to communicate from the source cluster over the @@ -26,35 +38,50 @@ 8092.

    From the XDCR menu, you can configure Remote - Clusters for XDCR; these are named destination clusters you can select when you - configure replication. When you configure XDCR, the destination cluster reference should point + Clusters for XDCR; these are named destination-clusters you can select when you + configure replication. When you configure XDCR, the destination-cluster reference should point to the IP address of one of the nodes in the destination cluster. Once it has the connection - with that one node, similar to how a Couchbase SDK does, it will receive the cluster topology + with that one node, it receives the cluster topology of that destination cluster.

    Ongoing Replications are the ones that are configured and operating currently. You can monitor the current configuration, current status, and the last time a - replication process was triggered for each configured replication.

    -

    -
    Create a Remote Cluster Reference -

    The cluster reference sets the cluster name, IP or hostname, and administrator credentials.

    -

    To add a new node to an existing cluster using the Web Console:

    -
      -
    1. From the Couchbase Web Console > XDCR > click Create - Remote Cluster.
      2. -
    2. Enter the following information and click Save.
      3. -
    -

    -

    To create the unidirectional replication (from cluster A to cluster B):

    -
      - -
    1. Verify that a destination bucket exists in the cluster where you will be replicating.
      2. -
    2. To check that a destination bucket exists, issue a REST API request using the following - syntax, GET HTTP method, and URI path: - curl GET -u admin:password http://ip.for.destination.cluster:8091/pools/default/buckets
      3. -
    3. From the XDCR >Remote Clusters panel, - click Create Remote Cluster.
      4. -
    4. Enter the following information for identifying and accessing the destination cluster:

      + replication process was triggered for each configured replication. Before any replications have + been configured, the screen appears as follows: +

      + +

      + +

      + +
      + + Add a Remote Cluster + + +

      + By adding a remote cluster, you establish a target for Cross Data-Center Replication. The replicated data + originates from your local cluster. +

      + +

      + Before proceeding, ensure that the remote cluster contains a suitable + destination bucket, in which the replicated data can be saved. To check the buckets on + the remote cluster, either use Couchbase Web Console, or use a REST API request such as the following: +

      + + curl GET -u admin:password http://ip.for.destination.cluster:8091/pools/default/buckets + +

      On the XDCR screen (accessible from the left-hand navigation bar of Couchbase Web Console), left-click + on the Add Remote Cluster button. The following dialog appears: +

      + + + + +

      + Enter the following information, to identify the destination cluster: +

      + @@ -75,28 +102,37 @@ The IP address or hostname of a node in the destination cluster. - Username of Remote Cluster - The Administrator username for the destination cluster. + Username for Remote Cluster + The Full, Cluster, or XDCR Administrator username for the destination cluster. Password - The Administrator password for the destination cluster. + The Full, Cluster, or XDCR Administrator password for the destination cluster. - Enable TLS Encryption - Select the check box to enable XDCR data encryption using SSL and then - define the encryption level. The options are: Half - (only encrypt password) and Full (TLS encrypt password - and data). If selected, copy the root certificate from the Security - > Root Certificate page and paste into the box. + Enable Secure Connection + Select the checkbox to enable a secure connection, then + define the encryption-level. The options are: Half + (encrypt password only) and Full (encrypt both password + and data). If you select Full, copy the root certificate from the Security + > Root Certificate page of the remote cluster, and paste this + into the editable text-field that appears immediately below + the radio buttons. If the remote cluster requires a client certificate to be presented, paste this certificate + and the corresponding client key into the two lower boxes, as indicated. See + Encryption on the Wire + for information on using client and root certificates. +
      -

      5. -
    5. Click Save to store the new reference to the destination cluster. - This cluster information is now available when configuring replication for the source - cluster.
    + +

    + Click Save to store the new reference to the destination cluster. + This reference can now be used in setting up Cross Data-Center Replication from the local + cluster. +

    +
    Edit Remote Clusters

    To update the advanced replication settings using the Web Console:

      @@ -114,7 +150,7 @@
    Create Replication -

    Full, Cluster, and Replication Administrators can create a replication between clusters +

    Full, Cluster, and XDCR Administrators can create a replication between clusters after creating references to the source and destination cluster.

    After you configure and start replication, view the current status and list of replications in the Ongoing Replications section.

    @@ -122,13 +158,36 @@
  • From the XDCR >Ongoing Replications panel, click Create Replication to configure a new XDCR replication. The Add Replication window is displayed where you can configure a new - replication from the source to the destination cluster.

    • + replication from the source to the destination cluster. + +

    + +

    + +
  • In the Replicate From Bucket, select a bucket from the current cluster to replicate.
  • In the section Remote Cluster, select a destination cluster.
  • Enter the bucket name in the Remote Bucket box.
    • + +
  • + Select an XDCR Protocol. The options are: + +
      + +
    • Version 1, which should be selected only if the Elasticsearch plug-in + is being used. This version uses the REST protocol +
      • + +
    • Version 2, which is the default, and should be selected in all cases where + the Elasticsearch plug-in is not being used. This version uses the Memcached + Binary protocol. +
      • + +
    + +
    • +
  • Select the Enable advanced filteringcheck box. This will allow you to specify the filtering expression while creating replication. For more details, see .
    • @@ -136,10 +195,14 @@ href="#topic_e5r_qpn_vs/xdcr-adv" format="dita"/> section.
  • Click Save to start the replication process.
    • + +

    + Note that if authentication-issues (due to, for example, the non-availability of an LDAP server) + occur on the remote cluster after the replication process has started, the process may fail. +

    XDCR Advanced Settings -

    XDCR advanced settings are internal settings available for configuration and those can be - updated.

    +

    These are as follows:

    @@ -153,15 +216,13 @@ - XDCR Protocol - The XDCR protocol defaults to version 2.
      -
    • Version 1 uses the REST protocol for replication. If the Elasticsearch - plug-in is used, choose version 1.
      • -
    • Version 2 uses memcached REST protocol for replication, a high-performance - mode that directly uses the memcached protocol on the destination nodes. - Choose version 2 when setting up a new replication with Couchbase Server 2.2 - or later.
      • -
    + XDCR Compression Type + Defines whether documents are to be compressed for XDCR, and if so, what compression type + is to be used. The default is None. The only compression type currently supported + is Snappy. Note that compression is only used in accordance with the bucket's compression + mode setting. For further information, see + Compression. +     XDCR Source Nozzles per Node @@ -170,7 +231,7 @@ Node.

    A small value of two or four is often sufficient. The default is two and the value range is 1-100. The CLI command for setting this value is provided in .

     @@ -253,6 +314,11 @@ XDCR Statistics Collection Interval Shows how often XDCR Statistics is updated. + + XDCR Network Usage Limit (MB/sec) + The upper limit for network usage during replication, per source node, specified in megabytes per + second. The default is 0, which means no limit is applied. + XDCR Logging Level The log level for the replication. It can be Error, @@ -264,8 +330,8 @@

    Configure XDCR Filtering -

    Full, Cluster, and Replication Administrators can set up filtering in XDCR.

    -

    The filtering expression is a regular expression for filtering keys that +

    Full, Cluster, and XDCR Administrators can set up filtering in XDCR.

    +

    The filtering expression is a regular expression for filtering keys that need to be transmitted from the source cluster to the destination cluster. It is set when creating the XDCR replication.

    Filtering expressions are currently implemented only for the document @@ -281,7 +347,7 @@ Create Replication Enable Advanced filtering .

    -

    You cannot change a filtering expression on an existing replication. @@ -462,7 +528,7 @@ parameter starting from 1.

    Pause or Resume Replication -

    Full, Cluster, and Replication Administrators can pause and resume XDCR +

    Full, Cluster, and XDCR Administrators can pause and resume XDCR replication.

    Pause and Resume Replication using the UI

    XDCR streams between the source and destination cluster can be paused and later resumed. @@ -486,12 +552,12 @@

    Pause and Resume Replication using the CLI

    To use the CLI, see .

    Pause and Resume Replication using the REST API

    + href="../cli/cbcli/couchbase-cli-xdcr-replicate.html" format="html"/>.

    Pause and Resume Replication using the REST API

    To use the REST API, see .

    Monitor Replication Status -

    Full, Cluster, Read-only, and Replication Administrators can monitor the replication +

    Full, Cluster, Read-only, and XDCR Administrators can monitor the replication status using the XDCR and Data Buckets tabs.

    The following Couchbase Web Console areas contain information about replication via XDCR: @@ -527,7 +593,7 @@

    Delete XDCR Replication -

    Full, Cluster, and Replication Administrators can delete active replications.

    +

    Full, Cluster, and XDCR Administrators can delete active replications.

    To delete the replication, delete the active replications using the Web Console:

    1. From the Couchbase Web Console > XDCR > Ongoing Replications, @@ -536,8 +602,61 @@
    2. Click Yes to confirm the deletion process.
    + +
    + + + XDCR via CLI + + +

    + You can manage XDCR by means of the Couchbase CLI: +

    + +
      +
    • + xdcr-setup manages references to remote clusters. + +

      + +

      +
      • + +
    • + xdcr-setup creates replications. + +

      + +

      +
      • + +
    • + setting-xdcr modifies replication-settings. + +

      + +

      +
      • + +
    + +
    + +
    + + + XDCR via REST + + +

    + You can manage XDCR by means of the Couchbase REST API. A full list of the endpoints is provided in + XDCR API. +

    + +
    + - + diff --git a/content/xdcr/xdcr-delete.dita b/content/xdcr/xdcr-delete.dita index 1df38fba26..07462bd698 100644 --- a/content/xdcr/xdcr-delete.dita +++ b/content/xdcr/xdcr-delete.dita @@ -11,6 +11,6 @@ - + diff --git a/content/xdcr/xdcr-managing-security.dita b/content/xdcr/xdcr-managing-security.dita index 38626e4f0e..c977afd066 100644 --- a/content/xdcr/xdcr-managing-security.dita +++ b/content/xdcr/xdcr-managing-security.dita @@ -1,76 +1,369 @@ + + - XDCR Data Security The data replicated between clusters can be encrypted in both uni-directional and bi-directional replications. Only the Full Administrator can manage XDCR data security settings. + + + XDCR Security + + + + XDCR security is configurable + + -

    When you use Secure Cross Datacenter Replication (XDCR), all traffic from the source and - destination data centers will be encrypted. Encryption causes a slight increase in the CPU - load since it requires additional CPU cycles.

    - -
    Enable XDCR Data Security - -

    To enable XDCR data security using SSL and create replication:

    + +
    + + + Overview + + +

    + By default, for inter-cluster communications, XDCR transmits both password and data in non-secure form. + Optionally, a secure connection + can be enabled between clusters, in order to secure either password alone, or both password and data. +

    + +

    + Note that + if the password received by the destination cluster requires authentication by an LDAP server, the destination + cluster communicates with the LDAP server in plain text, using saslauthd. This is described in + Setting Up saslauthd. +

    + +

    + A secure XDCR connection is enabled either by SCRAM-SHA or by TLS — depending on the + administrator-specified connection-type, + and the server-version of the destination cluster. + Use of TLS involves certificate management: for + information on preparing and using certificates, see + Certificate-Based Authentication. +

    + +
    + +
    + + + Start Enablement + + +

    + Proceed as follows: +

      -
    1. On the destination cluster, navigate to - Security - Root Certificate - and copy the certificate.

      (Optional) To regenerate the existing - destination certificate, click Regenerate before copying.

      2. -
    2. In the source cluster, select the XDCR tab.
      3. -
    3. In the Remote Clusters panel, click Create Cluster - Reference to verify or create the cluster reference.
      4. -
    4. Select the Enable TLS Encryption box and paste the certificate in - the provided area and click Save.

      5. -
    5. In the Ongoing Replications panel, click Create Replication, provide the cluster and bucket information, and click Replicate. - If this is an existing replication that is simply enabling TLS the replication will automatically restart to enable the SSL communication. During restart XDCR will use the last check point of the replication stream.
      6. + +
    6. + On the source cluster, left-click on the XDCR tab. This brings up + the XDCR Replications screen. + +

      + +

      +
      7. -
    -
    Change XDCR Data Encryption -

    In some situations, such as updating SSL data security, the SSL certificate is regenerated, and the XDCR data encryption is updated. Only the Full Administrator can regenerate the SSL certificate to update XDCR data - encryption.

    -

    To change XDCR data encryption:

    +
  • + Do one of the following: + +
      +
    • + To create a new cluster-reference, in the Remote Clusters panel, + left-click on the Add Remote Cluster button, to the right. This + brings up the Add Remote Cluster dialog. + +

      + +

      +
      • + +
    • + To edit an existing cluster-reference, in the Remote Clusters panel, + left-click on the Edit tab, at the right of the row of an + existing cluster-reference. This brings up + the Edit Remote Cluster dialog. + +

      + +

      + +
      • +
    + + In the dialog you have brought up, enter appropriate information for the Cluster Name and IP/Hostname. + Then, proceed as appropriate for the type of connection you wish to establish, according to the instructions below. + Note that in each example, the username and password specified must be those + of the Full, Cluster, or XDCR Administrator for the destination cluster. + +
    • + + + +
    + +
    + + + Enable a Non-Secure Connection + + +

    + In the open Add or Edit dialog, enter the + Username for Remote Cluster, and Password. Then, left-click + on the Save button, at the lower right of the dialog. +

    + +
    + +
    + + + Enable a 'Half' Secure Connection + + +

    + A Half secure connection secures the specified password only: it does not secure data. The + password is secured: +

    + +
      +
    • + By hashing with SCRAM-SHA, when the destination cluster is running Couchbase Enterprise Server 5.5 or later. + +

      + +

      +
      • + +
    • + By TLS encryption, when the destination cluster is running a pre-5.5 Couchbase Enterprise Server. +

      + +

      +
      • +
    + +

    + The root certificate of the destination cluster must be provided, for a successful TLS connection to + be achieved. If this certificate is provided and a SCRAM-SHA connection is achieved, the certificate + is ignored. The root certificate can be obtained by accessing the Root Certificate tab, on the + Security screen for the remote cluster: copy the certificate from the interactive panel in which + it appears. +

    + +

    + To enable a secure half connection with a destination cluster + that may be either pre-5.0 or 5.0 and beyond, + proceed as follows: +

    +
      -
    1. On the destination cluster, navigate to - Settings - Cluster - .
      2. -
    2. Click Regenerate and copy the certificate.
      3. -
    3. On the source cluster, select the XDCR tab.
      4. -
    4. In the Remote Clusters panel, for the destination cluster, click Edit.
      5. -
    5. Paste the regenerated certificate in the provided area and click Save. - Anytime you regenerate the destination cluster’s certificate, update the corresponding source cluster(s) - with that regenerated certificate. -

      For example, if source clusters A, B, and C use XDCR data encryption to replicate to destination cluster D, - update each of the source clusters whenever you regenerate the certificate on the destination cluster D.

      6. +
    6. + In the open Add or Edit dialog, enter the + Username for Remote Cluster, and Password. +

      + +

      +
      7. + +
    7. + Check the Enable Secure Connection checkbox. When the dialog expands vertically, select + the Half radio button. + +

      + +

      +
      8. + +
    8. + Copy and paste the root certificate for the destination cluster into the interactive panel, below + the radio buttons. The dialog now appears approximately as follows: + +

      + +

      + +
      9. + +
    9. + Left-click on the Save button, at the lower-right of the dialog. + +

      + +

      +
    - Replication will stop if you regenerate the destination cluster's - certificate and don't update the source cluster(s) with the new certificate.
    -
    Use an SSL Certificate -

    As a security best practice, - periodically rotate the XDCR certificates and also make sure that you instantiate a new certificate on the remote cluster.

    -

    The following example is a self-signed SSL/TLS certificate obtained on the cluster. Click - Security - Root Certificate - to reach the certificate.

    -

    -

    XDCR Data Security Error Messages

    -

    When creating the cluster reference, if the SSL certificates are not the same on the destination and source clusters, the following error message displays: + +

    + + + Enable a 'Full' Secure Connection + + +

    + A Full secure connection handles both authentication and data-transfer via + TLS. Select the + Full radio-button. When the dialog expands vertically to reveal two additional interactive panes, proceed in + one of the following ways, as appropriate. +

    + +
      +
    • + Specify username, password, and root certificate: + +
        +
      1. + In the open Add or Edit dialog, enter the + Username for Remote Cluster, and Password. +

        + +

        +
        2. + +
      2. + Copy and paste the root certificate for the destination cluster into the top interactive pane, below + the radio buttons — leaving both of the lower interactive panes blank. The dialog now appears + approximately as follows: + +

        + +

        +
        3. + +
      3. + Left-click on the Save button, at the lower-right of the dialog. + +

        + +

        +
        4. + +
      +
      • + +
    • + Specify root and client certificates, and client private key. + +

      + General information on preparing client certificates and keys is + provided in + X.509 for TLS. + Information on configuring clusters to handle client certificates is provided in + Handling Client Certificates. + Note that if the client certificate has been generated using the root certificate, the client certificate itself + must be specified. Alternatively, if the client certificate has been generated using intermediate certificates, the + entire certificate chain — including the client certificate and all intermediate certificates — must + be specified. +

      + +

      + Proceed as follows: +

      + +
        + +
      1. + Copy and paste the root certificate for the destination cluster into the top interactive pane. + +

        + +

        +
        2. + +
      2. + Copy and paste the client certificate for the local cluster into the middle interactive pane. + +

        + +

        +
        3. + +
      3. + Copy and paste the client private key for the local cluster into the bottom interactive pane. +

        + +

        + +
        4. + +
      4. + Ensure that the Username for Remote Cluster and Password + fields are blank. The dialog now appears approximately as follows: + +

        + +

        + +
        5. + +
      5. + Left-click on the Save button, at the lower-right of the dialog. + +

        + +

        +
        6. + +
      +
      • - Attention - Got certificate mismatch while trying to send https request to HOST:18091

      +
    +
    + +
    -

    The SSL certificates can become mismatched, such as when the certificate on the destination cluster is regenerated, - and the source cluster is not updated with the new certificate. - In this case, vBucket replication stops and the following error message displays: + + Starting Replication + - Error replicating vbucket <bucketNumber>. Please see logs for details.

    +

    + If you are adding a new cluster-reference, + in the Ongoing Replications panel, click Add Replication, + provide the cluster and bucket information, and click Replicate. This starts + replication. + +

    + +

    + Alternatively, if you are editing an existing replication, you do not have to take any further action: the existing + replication automatically restarts, with + TLS enabled. During restart, XDCR uses the last checkpoint of the replication stream. +

    + + +

    + Note that it is good practice + periodically to rotate XDCR certificates, and instantiate new ones. +

    + +
    + +
    + + + XDCR Security Error-Messages + + +

    + When creating the cluster reference, if certificates on the destination and source + clusters are mismatched, the following error message is displayed: Attention - Got certificate mismatch while trying to send https + request to HOST:18091. +

    + +

    + If XDCR is underway, and stops due to a certificate mismatch, the following error message is displayed: + Error replicating vbucket <bucketNumber>. Please see logs for details. +

    +
    + - + diff --git a/content/xdcr/xdcr-monitor-recovery.dita b/content/xdcr/xdcr-monitor-recovery.dita index bf4b88bd19..5179b67f64 100644 --- a/content/xdcr/xdcr-monitor-recovery.dita +++ b/content/xdcr/xdcr-monitor-recovery.dita @@ -40,6 +40,6 @@ msg is the number of documents recovered.

    - + diff --git a/content/xdcr/xdcr-monitor-status.dita b/content/xdcr/xdcr-monitor-status.dita index d2b3e0325c..926f788bac 100644 --- a/content/xdcr/xdcr-monitor-status.dita +++ b/content/xdcr/xdcr-monitor-status.dita @@ -33,6 +33,6 @@ - + diff --git a/content/xdcr/xdcr-pause-resume.dita b/content/xdcr/xdcr-pause-resume.dita index e341cfd9ab..f13f200f4f 100644 --- a/content/xdcr/xdcr-pause-resume.dita +++ b/content/xdcr/xdcr-pause-resume.dita @@ -32,12 +32,12 @@

    To use REST API or CLI:

    • REST API:
      • -
    • CLI: +
    • CLI:

    - + diff --git a/content/xdcr/xdcr-recover-partitions.dita b/content/xdcr/xdcr-recover-partitions.dita index 200bba6486..4e55450ec6 100644 --- a/content/xdcr/xdcr-recover-partitions.dita +++ b/content/xdcr/xdcr-recover-partitions.dita @@ -1,104 +1,223 @@ + - Recover Partitions from a Remote Cluster Data recovery from remote clusters requires XDCR and enough memory and disk - space. + + XDCR-Based Data Recovery + + + + In the event of data-loss, the cbrecovery tool can be used to restore data. The tool + accesses remotely replicated buckets, previously created with XDCR, and copies appropriate subsets of their data back + onto the original source-cluster. + + -

    If more nodes in a cluster catastrophically fail than the number of replicas, data partitions - in that cluster will no longer be available. For example, if you have a four-node cluster with - one replica and two nodes fail and are unrecoverable, some data shards will no longer be - available.

    -

    Data recovery from remote clusters requires an XDCR environment and adequate amount of memory and disk space to support the workload and recovered data. -

    -

    To recover the missing data partition in case of node failure and when vBuckets are unavailable, follow these steps. -

    -
      -
    1. For each failed node, click on - Server Nodes - Fail Over - .

      You will see whether data is unavailable and which vBuckets are - unavailable. If you do not have enough replicas for the number of the failed-over nodes, - some vBuckets will no longer be available:

      2. -
    2. Add new functioning nodes to replace the failed nodes.

      Do not rebalance after you add - new nodes to a cluster. Typically, you will rebalance after adding nodes to the cluster, - but in this scenario the rebalance will destroy the information about the missing vBuckets - and you cannot recover it.

      In this example, two nodes failed - in a three-node cluster and a new node at 10.3.3.61 was added. If you are - certain that your cluster can easily handle the workload and recovered data, you may - choose to skip this step.

      3. -
    3. Run cbrecovery to recover data from your backup cluster. In the - Server Panel, the Stop Recovery button - appears.

      After the recovery completes, this button disappears.

      4. -
    4. Rebalance your cluster.

      Once the recovery is completed, you can rebalance your cluster. - Rebalancing will recreate replica vBuckets and redistribute them evenly across the - cluster.

      -
      5. -
    -
    Recovery Dry-Run -

    Preview a list of buckets that are no longer available in the cluster.

    -

    Before you recover vBuckets, you may want to preview a list of buckets that are no longer - available in the cluster. Use this command and options:

    - shell> ./cbrecovery http://Administrator:password@10.3.3.72:8091 http://Administrator:password@10.3.3.61:8091 -n -

    The administrative credentials are provided for the node in the cluster, as well as the - option -n. The command returns a list of vBuckets in the remote secondary - cluster that are no longer in the first cluster. If there are any unavailable buckets in the - cluster with failed nodes, you will see output as follows:

    - 2013-04-29 18:16:54,384: MainThread Missing vbuckets to be recovered:[{"node": "ns_1@10.3.3.61", - "vbuckets": [513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526,, 528, 529, - 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545,, 547, 548, - 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, - 568, 569, 570, 571, 572,.... -

    In this case, the vbuckets array contains all the vBuckets that are no - longer available in the cluster. These are the buckets you can recover from the remotes - cluster using the following command:

    - shell> ./cbrecovery http://Administrator:password@<From_IP>:8091 \ - http://Administrator:password@<To_IP>:8091 -B bucket_name -

    You can run the command either on the cluster with unavailable vBuckets, or on the remote - cluster.

    -

    Provide the hostname, port, and credentials for the remote cluster and the cluster with - missing vBuckets, in that order. If you do not provide the parameter -B, the - tool assumes you will recover unavailable vBuckets for the default bucket.

    -
    Monitor the Recovery Process -

    Full, Cluster, Read-only, and Replication Administrators can use the Couchbase Web - Console to monitor the recovery process.

    -

    To monitor the progress of recovery: + +

    + + Understanding XDCR-Based Data Recovery + + +

    + Couchbase Server allows one or more replicas to be created for each vBucket on the cluster. This helps + to ensure continued data-availability in the event of node-failure.

    + +

    + However, if multiple nodes within a single cluster fail simultaneously, + one or more active vBuckets and all their replicas may be affected; meaning that lost data cannot be + recovered locally. +

    + +

    + In such cases, + provided that a bucket affected by such failure has already been established as a source bucket for XDCR, the + lost data may be retrieved from the bucket defined + on the remote server as the corresponding replication-target. This retrieval is achieved from the command-line, by means of + the Couchbase cbrecovery tool. +

    + +
    + +
    + + Recovering Lost Data + + +

    + The + illustration below shows a three-node cluster, named 10.142.180.101. +

    + +

    + +

    + +

    + The sample bucket travel-sample has been + established across this cluster, with one replica specified. + An XDCR reference to a remote cluster, named 10.142.180.104, + has also been established; and a replication started, such that travel-sample + is being replicated to the bucket travelSampleBackup, on the remote cluster. +

    + +

    + +

    + +

    + If two of the three nodes in the local cluster become unavailable, the Server screen for the local cluster + provides corresponding notifications: +

    + +

    + +

    + +

    + Since travel-sample was established with one replica, the permanent unavailability two nodes out of three + means data-loss from the local cluster. + To begin the XDCR-based recovery-process, proceed as follows. +

    +
      -
    1. Click on the Data Buckets tab.
      2. -
    2. Select the data bucket you are recovering from the Data Buckets drop-down list.
      3. -
    3. Click on the Summary drop-down list to see more details about this - data bucket. You will see an increased number of items during recovery:

      4. -
    4. You can also see the number of active vBuckets increase as they are recovered until you - reach 1024 vBuckets.

      Click on the vBucket Resources drop-down:

      -

      Since this tool runs from the command line, you can stop it at any - time.

      5. -
    5. The Stop Recovery button appears in the Servers - panels. If you click this button, you will stop the recovery process between clusters. Once - the recovery process completes, this button will no longer appear, and you will need to - rebalance the cluster. You can also stop it in this panel:

      6. -
    6. After the recovery completes, click on the Server Nodes tab and then on Rebalance to - rebalance your cluster. When cbrecovery finishes, it will output a report - in the console: - Recovery : Total | Per sec - batch : 0000 | 14.5 - byte : 0000 | 156.0 - msg : 0000 | 15.6 - 4 vbuckets recovered with elapsed time 10.90 seconds + +
    7. + If the unavailable nodes and their data absolutely cannot be retrieved, fail + each node over, by left-clicking the Failover button. Then, remove each node from + the cluster, by left-clicking the Remove button. + +

      +

      +
      8. + +
    8. + Before attempting to recover the lost data, restore capacity to the local cluster, + as appropriate. The + illustration below shows the two nodes that were previously lost, cleansed of all data, powered up, and in the + process of being re-added + into the cluster: + +

      + +

      + +

      + Do not at this point rebalance the cluster: the rebalance operation + affects local vBucket data, and thereby + prevents recovery of lost data from the remote cluster. Rebalance will be performed after the lost + data has been recovered. +

      + +

      + +

      +
      9. + +
    9. + Use the cbrecovery tool to restore data to the bucket travel-sample, from the bucket established + on the remote cluster, travelSampleBackup. + + $ cbrecovery http://10.142.180.104:8091 http://10.142.180.101:8091 \ +> -b travelSampleBackup \ +> -B travel-sample \ +> -u Administrator \ +> -p password \ +> -U Administrator \ +> -P password \ +> -v + +

      + For information on all parameter-options, see + cbrecovery. + Used, as shown here, with the verbose option, the command provides extensive console output. The initial portion + appears as follows: +

      + + Missing vbuckets to be recovered:[{"node": "ns_1@10.142.180.102", "vbuckets": [171, 172, +173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, +191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, +209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, +227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, +245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, +263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, +281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, +299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, +317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, +335, 336, 337, 338, 339, 340, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, +354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, +372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, +390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, +408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, +426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, +444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, +462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, +480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, +498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511]}] +2018-08-13 12:28:07,898: mt cbrecovery... +2018-08-13 12:28:07,898: mt source : http://10.142.180.104:8091 +2018-08-13 12:28:07,898: mt sink : http://10.142.180.101:8091 +2018-08-13 12:28:07,898: mt opts : {'username': '<xxx>', 'username_destination': +'Administrator', 'verbose': 1, 'extra': {'max_retry': 10.0, 'rehash': 0.0, +'dcp_consumer_queue_length': 1000.0, 'data_only': 1.0, 'uncompress': 0.0, +'nmv_retry': 1.0, 'conflict_resolve': 0.0, 'cbb_max_mb': 100000.0, 'report': 5.0, +'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, +'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, +'design_doc_only': 0.0, 'allow_recovery_vb_remap': 1.0, 'recv_min_bytes': 4096.0}, +'collection': None, 'ssl': False, 'threads': 4, 'key': None, 'password': '<xxx>', + 'id': None, 'silent': False, 'dry_run': False, 'password_destination': 'password', + 'bucket_destination': 'travel-sample', 'vbucket_list': '{"ns_1@10.142.180.102": [171]}', + 'separator': '::', 'bucket_source': 'travelSampleBackup'} +2018-08-13 12:28:07,939: mt Starting new HTTP connection (1): 10.142.180.104 +2018-08-13 12:28:07,992: mt Starting new HTTP connection (1): 10.142.180.101 +2018-08-13 12:28:08,010: mt bucket: travelSampleBackup +2018-08-13 12:28:08,229: w0 source : http://10.142.180.104:8091(travelSampleBackup@10.142.180.104:8091) +2018-08-13 12:28:08,229: w0 sink : http://10.142.180.101:8091(travelSampleBackup@10.142.180.104:8091) +2018-08-13 12:28:08,229: w0 : total | last | per sec +2018-08-13 12:28:08,229: w0 batch : 1 | 1 | 4.6 +2018-08-13 12:28:08,230: w0 byte : 21650 | 21650 | 99915.5 +2018-08-13 12:28:08,230: w0 msg : 23 | 23 | 106.1 +[ ] 0.1% (23/estimated 17018 msgs) +bucket: travelSampleBackup, msgs transferred... +: total | last | per sec +batch : 1 | 1 | 3.3 +byte : 21650 | 21650 | 71898.7 +msg : 23 | 23 | 76.4 +transfer data only. bucket design docs and index meta will be skipped. +done + +

      + Additionally, you can monitor the recovery process by means of the Couchbase Web Console: access the Statistics panel + for the travel-sample + bucket, on the local cluster; and inspect the contents of the Incoming XDCR Operations panel. +

      + +

      + When cbrecovery has concluded, a message similar to the following is displayed on the console: +

      + + Recovery : Total | Per sec +batch : 340 | 1.8 +byte : 9708874 | 51930.7 +msg : 8509 | 45.5 +340 vbuckets recovered with elapsed time 186.96 seconds + +
      10. + +
    10. + To conclude the data-recovery process, rebalance the nodes on the local cluster, by left-clicking the Rebalance + button. +

      + +

      +
    -

    In this report: batch is a group of internal operations performed by - cbrecovery, byte indicates the total number of bytes recovered, and - msg is the number of documents recovered.

    +
    + + diff --git a/content/xdcr/xdcr-tuning-performance.dita b/content/xdcr/xdcr-tuning-performance.dita index 97a8f651a9..404ab53d4b 100644 --- a/content/xdcr/xdcr-tuning-performance.dita +++ b/content/xdcr/xdcr-tuning-performance.dita @@ -78,6 +78,6 @@
    - +