2020-02-18 16:43:19 +00:00
# esquery
2020-02-23 10:29:45 +00:00
[](https://godoc.org/github.com/aquasecurity/esquery) [](LICENSE) [](https://travis-ci.org/aquasecurity/esquery)
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
2020-02-20 09:39:31 +00:00
**A non-obtrusive, idiomatic and easy-to-use query and aggregation builder for the [official Go client ](https://github.com/elastic/go-elasticsearch ) for [ElasticSearch ](https://www.elastic.co/products/elasticsearch ).**
## Table of Contents
<!-- ts -->
* [Description ](#description )
* [Status ](#status )
* [Installation ](#installation )
* [Usage ](#usage )
* [Notes ](#notes )
* [Features ](#features )
* [Supported Queries ](#supported-queries )
* [Supported Aggregations ](#supported-aggregations )
* [Custom Queries and Aggregations ](#custom-queries-and-aggregations )
* [License ](#license )
<!-- te -->
## Description
`esquery` alleviates the need to use extremely nested maps (`map[string]interface{}`) and serializing queries to JSON manually. It also helps eliminating common mistakes such as misspelling query types, as everything is statically typed.
Using `esquery` can make your code much easier to write, read and maintain, and significantly reduce the amount of code you write. Wanna know how much code you'll save? just check this project's tests.
## Status
This is an early release, API may still change.
## Installation
`esquery` is a Go module. To install, simply run this in your project's root directory:
```bash
go get github.com/aquasecurity/esquery
```
2020-02-18 16:43:19 +00:00
## Usage
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
esquery provides a [method chaining ](https://en.wikipedia.org/wiki/Method_chaining )-style API for building and executing queries and aggregations. It does not wrap the official Go client nor does it require you to change your existing code in order to integrate the library. Queries can be directly built with `esquery` , and executed by passing an `*elasticsearch.Client` instance (with optional search parameters). Results are returned as-is from the official client (e.g. `*esapi.Response` objects).
Getting started is extremely simple:
2020-02-18 16:43:19 +00:00
```go
package main
import (
"context"
"log"
2020-02-20 09:39:31 +00:00
"github.com/aquasecurity/esquery"
2020-02-18 16:43:19 +00:00
"github.com/elastic/go-elasticsearch/v7"
)
func main() {
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
// connect to an ElasticSearch instance
2020-04-06 09:03:42 +00:00
es, err := elasticsearch.NewDefaultClient()
if err != nil {
log.Fatalf("Failed creating client: %s", err)
}
2020-02-18 16:43:19 +00:00
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
// run a boolean search query
2020-04-06 09:03:42 +00:00
res, err := esquery.Search().
Query(
esquery.
Bool().
Must(esquery.Term("title", "Go and Stuff")).
Filter(esquery.Term("tag", "tech")),
).
Aggs(
esquery.Avg("average_score", "score"),
esquery.Max("max_score", "score"),
).
Size(20).
Run(
2021-03-14 07:50:22 +00:00
es,
2020-04-06 09:03:42 +00:00
es.Search.WithContext(context.TODO()),
es.Search.WithIndex("test"),
)
if err != nil {
log.Fatalf("Failed searching for stuff: %s", err)
}
defer res.Body.Close()
2020-02-18 16:43:19 +00:00
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
// ...
2020-02-18 16:43:19 +00:00
}
```
## Notes
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
* `esquery` currently supports version 7 of the ElasticSearch Go client.
2020-02-18 16:43:19 +00:00
* The library cannot currently generate "short queries". For example, whereas
ElasticSearch can accept this:
```json
{ "query": { "term": { "user": "Kimchy" } } }
```
The library will always generate this:
```json
{ "query": { "term": { "user": { "value": "Kimchy" } } } }
```
This is also true for queries such as "bool", where fields like "must" can
either receive one query object, or an array of query objects. `esquery` will
generate an array even if there's only one query object.
2020-02-20 09:39:31 +00:00
## Features
### Supported Queries
2020-02-18 16:43:19 +00:00
The following queries are currently supported:
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
| ElasticSearch DSL | `esquery` Function |
2020-02-18 16:43:19 +00:00
| ------------------------|---------------------- |
| `"match"` | `Match()` |
| `"match_bool_prefix"` | `MatchBoolPrefix()` |
| `"match_phrase"` | `MatchPhrase()` |
| `"match_phrase_prefix"` | `MatchPhrasePrefix()` |
| `"match_all"` | `MatchAll()` |
| `"match_none"` | `MatchNone()` |
2021-03-14 07:50:22 +00:00
| `"multi_match"` | `MultiMatch()` |
2020-02-18 16:43:19 +00:00
| `"exists"` | `Exists()` |
| `"fuzzy"` | `Fuzzy()` |
| `"ids"` | `IDs()` |
| `"prefix"` | `Prefix()` |
| `"range"` | `Range()` |
| `"regexp"` | `Regexp()` |
| `"term"` | `Term()` |
| `"terms"` | `Terms()` |
| `"terms_set"` | `TermsSet()` |
| `"wildcard"` | `Wildcard()` |
| `"bool"` | `Bool()` |
| `"boosting"` | `Boosting()` |
| `"constant_score"` | `ConstantScore()` |
| `"dis_max"` | `DisMax()` |
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
2020-02-20 09:39:31 +00:00
### Supported Aggregations
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
The following aggregations are currently supported:
| ElasticSearch DSL | `esquery` Function |
| ------------------------|---------------------- |
| `"avg"` | `Avg()` |
| `"weighted_avg"` | `WeightedAvg()` |
| `"cardinality"` | `Cardinality()` |
| `"max"` | `Max()` |
| `"min"` | `Min()` |
| `"sum"` | `Sum()` |
| `"value_count"` | `ValueCount()` |
| `"percentiles"` | `Percentiles()` |
| `"stats"` | `Stats()` |
| `"string_stats"` | `StringStats()` |
2020-04-06 09:03:42 +00:00
| `"top_hits"` | `TopHits()` |
| `"terms"` | `TermsAgg()` |
2022-01-21 06:15:54 +00:00
| `"date_histogram"` | `DateHistogramAgg()` |
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
2021-03-14 07:50:22 +00:00
### Supported Top Level Options
The following top level options are currently supported:
| ElasticSearch DSL | `esquery.Search` Function |
| ------------------------|--------------------------------------- |
| `"highlight"` | `Highlight()` |
| `"explain"` | `Explain()` |
| `"from"` | `From()` |
| `"postFilter"` | `PostFilter()` |
| `"query"` | `Query()` |
| `"aggs"` | `Aggs()` |
| `"size"` | `Size()` |
| `"sort"` | `Sort()` |
| `"source"` | `SourceIncludes(), SourceExcludes()` |
| `"timeout"` | `Timeout()` |
2020-02-20 09:39:31 +00:00
#### Custom Queries and Aggregations
To execute an arbitrary query or aggregation (including those not yet supported by the library), use the `CustomQuery()` or `CustomAgg()` functions, respectively. Both accept any `map[string]interface{}` value.
## License
Refactor API, add aggregations and custom queries
This commit introduces a refactor of the codebase and the API, to make
it more user friendly. Queries can now directly be executed via the
`Run()` method. Internally, the library no longer uses JSON generation
as a major mechanism, instead all types need to implement a `Mappable`
interface which simply turns each type in a `map[string]interface{}`,
which is what the ElasticSearch client expects. This makes the code
easier to write, and makes writing tests less error prone, as JSON need
not be written directly.
Support for metrics aggregations is also added. However, aggregations of
type bucket, pipeline and matrix are not supported yet.
To make the library more useful in its current state, support is added
for running custom queries and aggregations, via the `CustomQuery()` and
`CustomAgg()` functions, which both accepts an arbitrary
`map[string]interface{}`.
2020-02-19 11:35:21 +00:00
2020-02-20 09:39:31 +00:00
This library is distributed under the terms of the [Apache License 2.0 ](LICENSE ).
