ldetool
is a command line utility to generate Go code for fast log files parsing.
go install github.com/sirkon/ldetool@latest
<something>.lde
ldetool <something.lde> --package main
. Of course
you can use your own package name, not only main
Parse(line []byte)
.It turned out we like using it even for non-performant tasks, where we are dealing with strings, not slices of bytes and it would be handy to use it for strings as well without manual type casting. There’s an option to generate code that use string, just put an option
--go-string
--go-string
generates code that uses string
everywhere instead of []byte
. You better not to use it for log processing as it may lead to excessive memory allocations.--yaml-dict
or --json-dict
sets translation rules for names. For instance, if we have YAML file with
http: HTTP
and feed this file to the ldetool
then every name (of field or rule itself) like GetHttpHandle
or get_http_handle
will be translated into GetHTTPHandle
--package <pkg name>
name of the package to use in generated code. If a directory of *.lde
file has other Go files package name will automatically setup with these files’ package name.--big-endian
or --little-endian
sets the target architecture to be either big or little endian. This
enables prefix check optimizationTake a look at these two lines
[2017-09-02T22:48:13] FETCH first[1] format[JSON] hidden[0] userAgent[Android App v1.0] rnd[21341975] country[MA]
[2017-09-02T22:48:14] FETCH first[0] format[JSON] userAgent[Android App v1.0] rnd[10000000] country[LC]
We likely need a time, value of parameter first
, format
, hidden
, userAgent
and country
. We obviously don’t need rnd
See more details on extraction rules
# filename: line.lde
Line = # Name of the extraction object' type
^'[' Time(string) ']' # The line must start with [, then take everything as a struct field Time string right to ']' character
^" FETCH " # Current rest must starts with " FETCH " string
^"first[" First(uint8) ']'[1] # The rest must starts with "first[" characters, then take the rest until ']' as uint8. It is
# known First is the single character, thus the [1] index.
# under the name of First
^" format[" Format(string) ~']' # Take format id. Format is a short word: XML, JSON, BIN. ~ before lookup oobject suggests
# generator to use for loop scan rather than IndexByte, which is although fast
# has call overhead as it cannot be inlined by Go compiler.
?Hidden (^" hidden[" Value(uint8) ']') # Optionally look for " hidden[\d+]"
^" user_agent[" UserAgent(string) ']' # User agent data
_ "country[" Country(string) ']' # Look for the piece starting with country[
;
The recommended way is to put something like //go:generate ldetool --package main Line.lde
in generate.go
of a package and then generate a code with
go generate <project path>
It will be written into line_lde.go
file in the same directory. It will look like this
Now, we have
// Line autogenerated parser
type Line struct {
Rest []byte
Time []byte
First uint8
Format []byte
Hidden struct {
Valid bool
Value uint8
}
UserAgent []byte
Country []byte
}
// Extract autogenerated method of Line
func (p *Line) Extract(line []byte) (bool, error) {
…
}
Take a look at return data. First bool signals if the data was successfully matched and error that is not nil signals if there were
any error. String to numeric failures are always treated as errors, you can put !
into extraction script and all
mismatches after the sign will be treated as errors
Hidden
area returning default Go value if the the area was not matched
// GetHiddenValue retrieves optional value for HiddenValue.Name
func (p *Line) GetHiddenValue() (res uint8) {
if !p.Hidden.Valid {
return
}
return p.Hidden.Value
}
It is easy: put
l := &Line{}
before and then feed Extract
method with lines:
scanner := bufio.NewScanner(reader)
for scanner.Scan() {
ok, err := l.Extract(scanner.Bytes())
if !ok {
if err != nil {
return err
}
continue
}
…
l.Format
l.Time
l.GetHiddenValue()
…
}
Special thanks to Matt Hook (github.com/hookenz) who proposed this feature
It is possible to use custom types in generated structure. You should declare them first via
type pkg.Type from "pkgpath";
for external types and
type typeName;
for local types before all rules definitions and you can use them as field types. The parsing to be done via
p.unmarshal<FieldName>([]byte) (Type, error)
function.
Example:
type time.Time from "time";
type net.IP from "net";
Custom = Time(time.Time) ' ' ?Addr(^"addr: " IP(ip.IP) ' ');
Now, two parsing functions will be needed to parse this (they are to be written manually):
func (p *Custom) unmarshalTime(s string) (time.Time, error) { … }
func (p *Custom) unmarshalAddrIP(s string) (net.IP, error) { … }