mindoc-org
/
mindoc
mirror of https://github.com/mindoc-org/mindoc.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

更新依赖

pull/25/merge
Minho 2017-05-04 13:17:25 +08:00
parent 731a1e249f
commit 2298822933
374 changed files with 839 additions and 156772 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

118
Godeps/Godeps.json generated
View File

@ -2,14 +2,7 @@
"ImportPath": "github.com/lifei6671/godoc",
"GoVersion": "go1.8",
"GodepVersion": "v79",
"Packages": [
"./..."
],
"Deps": [
{
"ImportPath": "github.com/adamzy/cedar-go",
"Rev": "d348c21f72432c2b6d5f05f68759fde94f64b227"
},
{
"ImportPath": "github.com/astaxie/beego",
"Comment": "v1.8.1",
@ -45,6 +38,21 @@
"Comment": "v1.8.1",
"Rev": "323a1c4214101331a4b71922c23d19b7409ac71f"
},
{
"ImportPath": "github.com/astaxie/beego/session/memcache",
"Comment": "v1.8.0",
"Rev": "323a1c4214101331a4b71922c23d19b7409ac71f"
},
{
"ImportPath": "github.com/astaxie/beego/session/mysql",
"Comment": "v1.8.0",
"Rev": "323a1c4214101331a4b71922c23d19b7409ac71f"
},
{
"ImportPath": "github.com/astaxie/beego/session/redis",
"Comment": "v1.8.0",
"Rev": "323a1c4214101331a4b71922c23d19b7409ac71f"
},
{
"ImportPath": "github.com/astaxie/beego/toolbox",
"Comment": "v1.8.1",
@ -56,53 +64,19 @@
"Rev": "323a1c4214101331a4b71922c23d19b7409ac71f"
},
{
"ImportPath": "github.com/boltdb/bolt",
"Comment": "v1.3.0-58-ge9cf4fa",
"Rev": "e9cf4fae01b5a8ff89d0ec6b32f0d9c9f79aefdd"
"ImportPath": "github.com/bradfitz/gomemcache/memcache",
"Comment": "release.r60-46-g1952afa",
"Rev": "1952afaa557dc08e8e0d89eafab110fb501c1a2b"
},
{
"ImportPath": "github.com/cznic/fileutil",
"Rev": "90cf820aafe8f7df39416fdbb932029ff99bd1ab"
"ImportPath": "github.com/garyburd/redigo/internal",
"Comment": "v1.0.0-23-gac91d6f",
"Rev": "ac91d6ff49bd0d278a90201de77a4f8ad9628e25"
},
{
"ImportPath": "github.com/cznic/internal/buffer",
"Comment": "1.0.0-1-ge5e1c3e",
"Rev": "e5e1c3e9165d0a72507c2bbb0ffac1c02b8d3f7c"
},
{
"ImportPath": "github.com/cznic/internal/file",
"Comment": "1.0.0-1-ge5e1c3e",
"Rev": "e5e1c3e9165d0a72507c2bbb0ffac1c02b8d3f7c"
},
{
"ImportPath": "github.com/cznic/internal/slice",
"Comment": "1.0.0-1-ge5e1c3e",
"Rev": "e5e1c3e9165d0a72507c2bbb0ffac1c02b8d3f7c"
},
{
"ImportPath": "github.com/cznic/kv",
"Rev": "c5de474a2ccdaed5ba5ff8b5d2d213dbf48a8b5e"
},
{
"ImportPath": "github.com/cznic/lldb",
"Comment": "v1.1.0",
"Rev": "bea8611dd5c407f3c5eab9f9c68e887a27dc6f0e"
},
{
"ImportPath": "github.com/cznic/mathutil",
"Rev": "1447ad269d64ca91aa8d7079baa40b6fc8b965e7"
},
{
"ImportPath": "github.com/cznic/sortutil",
"Rev": "4c7342852e65c2088c981288f2c5610d10b9f7f4"
},
{
"ImportPath": "github.com/cznic/zappy",
"Rev": "2533cb5b45cc6c07421468ce262899ddc9d53fb7"
},
{
"ImportPath": "github.com/edsrzf/mmap-go",
"Rev": "0bce6a6887123b67a60366d2c9fe2dfb74289d2e"
"ImportPath": "github.com/garyburd/redigo/redis",
"Comment": "v1.0.0-23-gac91d6f",
"Rev": "ac91d6ff49bd0d278a90201de77a4f8ad9628e25"
},
{
"ImportPath": "github.com/go-sql-driver/mysql",
@ -110,45 +84,35 @@
"Rev": "1421caf44f6464fd2ee8de694c7508ee13f92964"
},
{
"ImportPath": "github.com/huichen/murmur",
"Rev": "e0489551cf5116e27d7cc69d97a53cdbdd028acf"
"ImportPath": "github.com/golang/freetype",
"Comment": "release-132-gd9be45a",
"Rev": "d9be45aaf7452cc30c0ceb1b1bf7efe1d17b7c87"
},
{
"ImportPath": "github.com/huichen/sego",
"Rev": "d06fe1b3abe3877ab593b57e5e43daf6c4c25add"
"ImportPath": "github.com/golang/freetype/raster",
"Comment": "release-132-gd9be45a",
"Rev": "d9be45aaf7452cc30c0ceb1b1bf7efe1d17b7c87"
},
{
"ImportPath": "github.com/huichen/wukong/core",
"Comment": "v0.1-94-gd014a1f",
"Rev": "d014a1f19dae3664677c11bd25549cfc820cd890"
"ImportPath": "github.com/golang/freetype/truetype",
"Comment": "release-132-gd9be45a",
"Rev": "d9be45aaf7452cc30c0ceb1b1bf7efe1d17b7c87"
},
{
"ImportPath": "github.com/huichen/wukong/engine",
"Comment": "v0.1-94-gd014a1f",
"Rev": "d014a1f19dae3664677c11bd25549cfc820cd890"
},
{
"ImportPath": "github.com/huichen/wukong/storage",
"Comment": "v0.1-94-gd014a1f",
"Rev": "d014a1f19dae3664677c11bd25549cfc820cd890"
},
{
"ImportPath": "github.com/huichen/wukong/types",
"Comment": "v0.1-94-gd014a1f",
"Rev": "d014a1f19dae3664677c11bd25549cfc820cd890"
},
{
"ImportPath": "github.com/huichen/wukong/utils",
"Comment": "v0.1-94-gd014a1f",
"Rev": "d014a1f19dae3664677c11bd25549cfc820cd890"
"ImportPath": "github.com/lifei6671/gocaptcha",
"Rev": "b244e66362327cfd7d47c1a75eb1ca3845ffc52a"
},
{
"ImportPath": "github.com/nfnt/resize",
"Rev": "891127d8d1b52734debe1b3c3d7e747502b6c366"
},
{
"ImportPath": "golang.org/x/sys/unix",
"Rev": "9ccfe848b9db8435a24c424abbc07a921adf1df5"
"ImportPath": "golang.org/x/image/font",
"Rev": "69afd001f792d732a78bd7225793315a8deb09ea"
},
{
"ImportPath": "golang.org/x/image/math/fixed",
"Rev": "69afd001f792d732a78bd7225793315a8deb09ea"
}
]
}

339
vendor/github.com/adamzy/cedar-go/LICENSE.md generated vendored
View File

@ -1,339 +0,0 @@
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc., <http://fsf.org/>
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
{description}
Copyright (C) {year} {fullname}
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
{signature of Ty Coon}, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License.

83
vendor/github.com/adamzy/cedar-go/README.md generated vendored
View File

@ -1,83 +0,0 @@
# cedar-go [![GoDoc](https://godoc.org/github.com/adamzy/cedar-go?status.svg)](https://godoc.org/github.com/adamzy/cedar-go)
Package `cedar-go` implementes double-array trie.
It is a [Golang](https://golang.org/) port of [cedar](http://www.tkl.iis.u-tokyo.ac.jp/~ynaga/cedar) which is written in C++ by Naoki Yoshinaga. `cedar-go` currently implements the `reduced` verion of cedar.
This package is not thread safe if there is one goroutine doing insertions or deletions.
## Install
```
go get github.com/adamzy/cedar-go
```
## Usage
```go
package main
import (
"fmt"
"github.com/adamzy/cedar-go"
)
func main() {
// create a new cedar trie.
trie := cedar.New()
// a helper function to print the id-key-value triple given trie node id
printIdKeyValue := func(id int) {
// the key of node `id`.
key, _ := trie.Key(id)
// the value of node `id`.
value, _ := trie.Value(id)
fmt.Printf("%d\t%s:%v\n", id, key, value)
}
// Insert key-value pairs.
// The order of insertion is not important.
trie.Insert([]byte("How many"), 0)
trie.Insert([]byte("How many loved"), 1)
trie.Insert([]byte("How many loved your moments"), 2)
trie.Insert([]byte("How many loved your moments of glad grace"), 3)
trie.Insert([]byte("姑苏"), 4)
trie.Insert([]byte("姑苏城外"), 5)
trie.Insert([]byte("姑苏城外寒山寺"), 6)
// Get the associated value of a key directly.
value, _ := trie.Get([]byte("How many loved your moments of glad grace"))
fmt.Println(value)
// Or, jump to the node first,
id, _ := trie.Jump([]byte("How many loved your moments"), 0)
// then get the key and the value
printIdKeyValue(id)
fmt.Println("\nPrefixMatch\nid\tkey:value")
for _, id := range trie.PrefixMatch([]byte("How many loved your moments of glad grace"), 0) {
printIdKeyValue(id)
}
fmt.Println("\nPrefixPredict\nid\tkey:value")
for _, id := range trie.PrefixPredict([]byte("姑苏"), 0) {
printIdKeyValue(id)
}
}
```
will produce
```
3
281 How many loved your moments:2
PrefixMatch
id key:value
262 How many:0
268 How many loved:1
281 How many loved your moments:2
296 How many loved your moments of glad grace:3
PrefixPredict
id key:value
303 姑苏:4
309 姑苏城外:5
318 姑苏城外寒山寺:6
```

231
vendor/github.com/adamzy/cedar-go/api.go generated vendored
View File

@ -1,231 +0,0 @@
package cedar
// Status reports the following statistics of the cedar:
// keys: number of keys that are in the cedar,
// nodes: number of trie nodes (slots in the base array) has been taken,
// size: the size of the base array used by the cedar,
// capacity: the capicity of the base array used by the cedar.
func (da *Cedar) Status() (keys, nodes, size, capacity int) {
for i := 0; i < da.Size; i++ {
n := da.Array[i]
if n.Check >= 0 {
nodes++
if n.Value >= 0 {
keys++
}
}
}
return keys, nodes, da.Size, da.Capacity
}
// Jump travels from a node `from` to another node `to` by following the path `path`.
// For example, if the following keys were inserted:
// id key
// 19 abc
// 23 ab
// 37 abcd
// then
// Jump([]byte("ab"), 0) = 23, nil // reach "ab" from root
// Jump([]byte("c"), 23) = 19, nil // reach "abc" from "ab"
// Jump([]byte("cd"), 23) = 37, nil // reach "abcd" from "ab"
func (da *Cedar) Jump(path []byte, from int) (to int, err error) {
for _, b := range path {
if da.Array[from].Value >= 0 {
return from, ErrNoPath
}
to = da.Array[from].base() ^ int(b)
if da.Array[to].Check != from {
return from, ErrNoPath
}
from = to
}
return to, nil
}
// Key returns the key of the node with the given `id`.
// It will return ErrNoPath, if the node does not exist.
func (da *Cedar) Key(id int) (key []byte, err error) {
for id > 0 {
from := da.Array[id].Check
if from < 0 {
return nil, ErrNoPath
}
if char := byte(da.Array[from].base() ^ id); char != 0 {
key = append(key, char)
}
id = from
}
if id != 0 || len(key) == 0 {
return nil, ErrInvalidKey
}
for i := 0; i < len(key)/2; i++ {
key[i], key[len(key)-i-1] = key[len(key)-i-1], key[i]
}
return key, nil
}
// Value returns the value of the node with the given `id`.
// It will return ErrNoValue, if the node does not have a value.
func (da *Cedar) Value(id int) (value int, err error) {
value = da.Array[id].Value
if value >= 0 {
return value, nil
}
to := da.Array[id].base()
if da.Array[to].Check == id && da.Array[to].Value >= 0 {
return da.Array[to].Value, nil
}
return 0, ErrNoValue
}
// Insert adds a key-value pair into the cedar.
// It will return ErrInvalidValue, if value < 0 or >= ValueLimit.
func (da *Cedar) Insert(key []byte, value int) error {
if value < 0 || value >= ValueLimit {
return ErrInvalidValue
}
p := da.get(key, 0, 0)
*p = value
return nil
}
// Update increases the value associated with the `key`.
// The `key` will be inserted if it is not in the cedar.
// It will return ErrInvalidValue, if the updated value < 0 or >= ValueLimit.
func (da *Cedar) Update(key []byte, value int) error {
p := da.get(key, 0, 0)
if *p+value < 0 || *p+value >= ValueLimit {
return ErrInvalidValue
}
*p += value
return nil
}
// Delete removes a key-value pair from the cedar.
// It will return ErrNoPath, if the key has not been added.
func (da *Cedar) Delete(key []byte) error {
// if the path does not exist, or the end is not a leaf, nothing to delete
to, err := da.Jump(key, 0)
if err != nil {
return ErrNoPath
}
if da.Array[to].Value < 0 {
base := da.Array[to].base()
if da.Array[base].Check == to {
to = base
}
}
for {
from := da.Array[to].Check
base := da.Array[from].base()
label := byte(to ^ base)
// if `to` has sibling, remove `to` from the sibling list, then stop
if da.Ninfos[to].Sibling != 0 || da.Ninfos[from].Child != label {
// delete the label from the child ring first
da.popSibling(from, base, label)
// then release the current node `to` to the empty node ring
da.pushEnode(to)
break
}
// otherwise, just release the current node `to` to the empty node ring
da.pushEnode(to)
// then check its parent node
to = from
}
return nil
}
// Get returns the value associated with the given `key`.
// It is equivalent to
// id, err1 = Jump(key)
// value, err2 = Value(id)
// Thus, it may return ErrNoPath or ErrNoValue,
func (da *Cedar) Get(key []byte) (value int, err error) {
to, err := da.Jump(key, 0)
if err != nil {
return 0, err
}
return da.Value(to)
}
// PrefixMatch returns a list of at most `num` nodes which match the prefix of the key.
// If `num` is 0, it returns all matches.
// For example, if the following keys were inserted:
// id key
// 19 abc
// 23 ab
// 37 abcd
// then
// PrefixMatch([]byte("abc"), 1) = [ 23 ] // match ["ab"]
// PrefixMatch([]byte("abcd"), 0) = [ 23, 19, 37] // match ["ab", "abc", "abcd"]
func (da *Cedar) PrefixMatch(key []byte, num int) (ids []int) {
for from, i := 0, 0; i < len(key); i++ {
to, err := da.Jump(key[i:i+1], from)
if err != nil {
break
}
if _, err := da.Value(to); err == nil {
ids = append(ids, to)
num--
if num == 0 {
return
}
}
from = to
}
return
}
// PrefixPredict returns a list of at most `num` nodes which has the key as their prefix.
// These nodes are ordered by their keys.
// If `num` is 0, it returns all matches.
// For example, if the following keys were inserted:
// id key
// 19 abc
// 23 ab
// 37 abcd
// then
// PrefixPredict([]byte("ab"), 2) = [ 23, 19 ] // predict ["ab", "abc"]
// PrefixPredict([]byte("ab"), 0) = [ 23, 19, 37 ] // predict ["ab", "abc", "abcd"]
func (da *Cedar) PrefixPredict(key []byte, num int) (ids []int) {
root, err := da.Jump(key, 0)
if err != nil {
return
}
for from, err := da.begin(root); err == nil; from, err = da.next(from, root) {
ids = append(ids, from)
num--
if num == 0 {
return
}
}
return
}
func (da *Cedar) begin(from int) (to int, err error) {
for c := da.Ninfos[from].Child; c != 0; {
to = da.Array[from].base() ^ int(c)
c = da.Ninfos[to].Child
from = to
}
if da.Array[from].base() > 0 {
return da.Array[from].base(), nil
}
return from, nil
}
func (da *Cedar) next(from int, root int) (to int, err error) {
c := da.Ninfos[from].Sibling
for c == 0 && from != root && da.Array[from].Check >= 0 {
from = da.Array[from].Check
c = da.Ninfos[from].Sibling
}
if from == root {
return 0, ErrNoPath
}
from = da.Array[da.Array[from].Check].base() ^ int(c)
return da.begin(from)
}

407
vendor/github.com/adamzy/cedar-go/cedar.go generated vendored
View File

@ -1,407 +0,0 @@
package cedar
const ValueLimit = int(^uint(0) >> 1)
type node struct {
Value int
Check int
}
func (n *node) base() int { return -(n.Value + 1) }
type ninfo struct {
Sibling, Child byte
}
type block struct {
Prev, Next, Num, Reject, Trial, Ehead int
}
func (b *block) init() {
b.Num = 256
b.Reject = 257
}
type Cedar struct {
*cedar
}
type cedar struct {
Array []node
Ninfos []ninfo
Blocks []block
Reject [257]int
BheadF int
BheadC int
BheadO int
Capacity int
Size int
Ordered bool
MaxTrial int
}
func New() *Cedar {
da := cedar{
Array: make([]node, 256),
Ninfos: make([]ninfo, 256),
Blocks: make([]block, 1),
Capacity: 256,
Size: 256,
Ordered: true,
MaxTrial: 1,
}
da.Array[0] = node{-2, 0}
for i := 1; i < 256; i++ {
da.Array[i] = node{-(i - 1), -(i + 1)}
}
da.Array[1].Value = -255
da.Array[255].Check = -1
da.Blocks[0].Ehead = 1
da.Blocks[0].init()
for i := 0; i <= 256; i++ {
da.Reject[i] = i + 1
}
return &Cedar{&da}
}
// Get value by key, insert the key if not exist
func (da *cedar) get(key []byte, from, pos int) *int {
for ; pos < len(key); pos++ {
if value := da.Array[from].Value; value >= 0 && value != ValueLimit {
to := da.follow(from, 0)
da.Array[to].Value = value
}
from = da.follow(from, key[pos])
}
to := from
if da.Array[from].Value < 0 {
to = da.follow(from, 0)
}
return &da.Array[to].Value
}
func (da *cedar) follow(from int, label byte) int {
base := da.Array[from].base()
to := base ^ int(label)
if base < 0 || da.Array[to].Check < 0 {
hasChild := false
if base >= 0 {
hasChild = (da.Array[base^int(da.Ninfos[from].Child)].Check == from)
}
to = da.popEnode(base, label, from)
da.pushSibling(from, to^int(label), label, hasChild)
} else if da.Array[to].Check != from {
to = da.resolve(from, base, label)
} else if da.Array[to].Check == from {
} else {
panic("cedar: internal error, should not be here")
}
return to
}
func (da *cedar) popBlock(bi int, head_in *int, last bool) {
if last {
*head_in = 0
} else {
b := &da.Blocks[bi]
da.Blocks[b.Prev].Next = b.Next
da.Blocks[b.Next].Prev = b.Prev
if bi == *head_in {
*head_in = b.Next
}
}
}
func (da *cedar) pushBlock(bi int, head_out *int, empty bool) {
b := &da.Blocks[bi]
if empty {
*head_out, b.Prev, b.Next = bi, bi, bi
} else {
tail_out := &da.Blocks[*head_out].Prev
b.Prev = *tail_out
b.Next = *head_out
*head_out, *tail_out, da.Blocks[*tail_out].Next = bi, bi, bi
}
}
func (da *cedar) addBlock() int {
if da.Size == da.Capacity {
da.Capacity *= 2
oldArray := da.Array
da.Array = make([]node, da.Capacity)
copy(da.Array, oldArray)
oldNinfo := da.Ninfos
da.Ninfos = make([]ninfo, da.Capacity)
copy(da.Ninfos, oldNinfo)
oldBlock := da.Blocks
da.Blocks = make([]block, da.Capacity>>8)
copy(da.Blocks, oldBlock)
}
da.Blocks[da.Size>>8].init()
da.Blocks[da.Size>>8].Ehead = da.Size
da.Array[da.Size] = node{-(da.Size + 255), -(da.Size + 1)}
for i := da.Size + 1; i < da.Size+255; i++ {
da.Array[i] = node{-(i - 1), -(i + 1)}
}
da.Array[da.Size+255] = node{-(da.Size + 254), -da.Size}
da.pushBlock(da.Size>>8, &da.BheadO, da.BheadO == 0)
da.Size += 256
return da.Size>>8 - 1
}
func (da *cedar) transferBlock(bi int, head_in, head_out *int) {
da.popBlock(bi, head_in, bi == da.Blocks[bi].Next)
da.pushBlock(bi, head_out, *head_out == 0 && da.Blocks[bi].Num != 0)
}
func (da *cedar) popEnode(base int, label byte, from int) int {
e := base ^ int(label)
if base < 0 {
e = da.findPlace()
}
bi := e >> 8
n := &da.Array[e]
b := &da.Blocks[bi]
b.Num--
if b.Num == 0 {
if bi != 0 {
da.transferBlock(bi, &da.BheadC, &da.BheadF)
}
} else {
da.Array[-n.Value].Check = n.Check
da.Array[-n.Check].Value = n.Value
if e == b.Ehead {
b.Ehead = -n.Check
}
if bi != 0 && b.Num == 1 && b.Trial != da.MaxTrial {
da.transferBlock(bi, &da.BheadO, &da.BheadC)
}
}
n.Value = ValueLimit
n.Check = from
if base < 0 {
da.Array[from].Value = -(e ^ int(label)) - 1
}
return e
}
func (da *cedar) pushEnode(e int) {
bi := e >> 8
b := &da.Blocks[bi]
b.Num++
if b.Num == 1 {
b.Ehead = e
da.Array[e] = node{-e, -e}
if bi != 0 {
da.transferBlock(bi, &da.BheadF, &da.BheadC)
}
} else {
prev := b.Ehead
next := -da.Array[prev].Check
da.Array[e] = node{-prev, -next}
da.Array[prev].Check = -e
da.Array[next].Value = -e
if b.Num == 2 || b.Trial == da.MaxTrial {
if bi != 0 {
da.transferBlock(bi, &da.BheadC, &da.BheadO)
}
}
b.Trial = 0
}
if b.Reject < da.Reject[b.Num] {
b.Reject = da.Reject[b.Num]
}
da.Ninfos[e] = ninfo{}
}
// hasChild: wherether the `from` node has children
func (da *cedar) pushSibling(from, base int, label byte, hasChild bool) {
c := &da.Ninfos[from].Child
keepOrder := *c == 0
if da.Ordered {
keepOrder = label > *c
}
if hasChild && keepOrder {
c = &da.Ninfos[base^int(*c)].Sibling
for da.Ordered && *c != 0 && *c < label {
c = &da.Ninfos[base^int(*c)].Sibling
}
}
da.Ninfos[base^int(label)].Sibling = *c
*c = label
}
func (da *cedar) popSibling(from, base int, label byte) {
c := &da.Ninfos[from].Child
for *c != label {
c = &da.Ninfos[base^int(*c)].Sibling
}
*c = da.Ninfos[base^int(*c)].Sibling
}
func (da *cedar) consult(base_n, base_p int, c_n, c_p byte) bool {
c_n = da.Ninfos[base_n^int(c_n)].Sibling
c_p = da.Ninfos[base_p^int(c_p)].Sibling
for c_n != 0 && c_p != 0 {
c_n = da.Ninfos[base_n^int(c_n)].Sibling
c_p = da.Ninfos[base_p^int(c_p)].Sibling
}
return c_p != 0
}
func (da *cedar) setChild(base int, c byte, label byte, flag bool) []byte {
child := make([]byte, 0, 257)
if c == 0 {
child = append(child, c)
c = da.Ninfos[base^int(c)].Sibling
}
if da.Ordered {
for c != 0 && c <= label {
child = append(child, c)
c = da.Ninfos[base^int(c)].Sibling
}
}
if flag {
child = append(child, label)
}
for c != 0 {
child = append(child, c)
c = da.Ninfos[base^int(c)].Sibling
}
return child
}
func (da *cedar) findPlace() int {
if da.BheadC != 0 {
return da.Blocks[da.BheadC].Ehead
}
if da.BheadO != 0 {
return da.Blocks[da.BheadO].Ehead
}
return da.addBlock() << 8
}
func (da *cedar) findPlaces(child []byte) int {
bi := da.BheadO
if bi != 0 {
bz := da.Blocks[da.BheadO].Prev
nc := len(child)
for {
b := &da.Blocks[bi]
if b.Num >= nc && nc < b.Reject {
for e := b.Ehead; ; {
base := e ^ int(child[0])
for i := 0; da.Array[base^int(child[i])].Check < 0; i++ {
if i == len(child)-1 {
b.Ehead = e
return e
}
}
e = -da.Array[e].Check
if e == b.Ehead {
break
}
}
}
b.Reject = nc
if b.Reject < da.Reject[b.Num] {
da.Reject[b.Num] = b.Reject
}
bi_ := b.Next
b.Trial++
if b.Trial == da.MaxTrial {
da.transferBlock(bi, &da.BheadO, &da.BheadC)
}
if bi == bz {
break
}
bi = bi_
}
}
return da.addBlock() << 8
}
func (da *cedar) resolve(from_n, base_n int, label_n byte) int {
to_pn := base_n ^ int(label_n)
from_p := da.Array[to_pn].Check
base_p := da.Array[from_p].base()
flag := da.consult(base_n, base_p, da.Ninfos[from_n].Child, da.Ninfos[from_p].Child)
var children []byte
if flag {
children = da.setChild(base_n, da.Ninfos[from_n].Child, label_n, true)
} else {
children = da.setChild(base_p, da.Ninfos[from_p].Child, 255, false)
}
var base int
if len(children) == 1 {
base = da.findPlace()
} else {
base = da.findPlaces(children)
}
base ^= int(children[0])
var from int
var base_ int
if flag {
from = from_n
base_ = base_n
} else {
from = from_p
base_ = base_p
}
if flag && children[0] == label_n {
da.Ninfos[from].Child = label_n
}
da.Array[from].Value = -base - 1
for i := 0; i < len(children); i++ {
to := da.popEnode(base, children[i], from)
to_ := base_ ^ int(children[i])
if i == len(children)-1 {
da.Ninfos[to].Sibling = 0
} else {
da.Ninfos[to].Sibling = children[i+1]
}
if flag && to_ == to_pn { // new node has no child
continue
}
n := &da.Array[to]
n_ := &da.Array[to_]
n.Value = n_.Value
if n.Value < 0 && children[i] != 0 {
// this node has children, fix their check
c := da.Ninfos[to_].Child
da.Ninfos[to].Child = c
da.Array[n.base()^int(c)].Check = to
c = da.Ninfos[n.base()^int(c)].Sibling
for c != 0 {
da.Array[n.base()^int(c)].Check = to
c = da.Ninfos[n.base()^int(c)].Sibling
}
}
if !flag && to_ == from_n { // parent node moved
from_n = to
}
if !flag && to_ == to_pn {
da.pushSibling(from_n, to_pn^int(label_n), label_n, true)
da.Ninfos[to_].Child = 0
n_.Value = ValueLimit
n_.Check = from_n
} else {
da.pushEnode(to_)
}
}
if flag {
return base ^ int(label_n)
}
return to_pn
}

12
vendor/github.com/adamzy/cedar-go/doc.go generated vendored
View File

@ -1,12 +0,0 @@
// Package cedar-go implements double-array trie.
//
// It is a golang port of cedar (http://www.tkl.iis.u-tokyo.ac.jp/~ynaga/cedar) which is written in C++ by Naoki Yoshinaga.
// Currently cedar-go implements the `reduced` verion of cedar.
// This package is not thread safe if there is one goroutine doing
// insertions or deletions.
//
// Note
//
// key must be `[]byte` without zero items,
// while value must be integer in the range [0, 2<<63-2] or [0, 2<<31-2] depends on the platform.
package cedar

11
vendor/github.com/adamzy/cedar-go/errors.go generated vendored
View File

@ -1,11 +0,0 @@
package cedar
import "errors"
var (
ErrInvalidDataType = errors.New("cedar: invalid datatype")
ErrInvalidValue = errors.New("cedar: invalid value")
ErrInvalidKey = errors.New("cedar: invalid key")
ErrNoPath = errors.New("cedar: no path")
ErrNoValue = errors.New("cedar: no value")
)

63
vendor/github.com/adamzy/cedar-go/io.go generated vendored
View File

@ -1,63 +0,0 @@
package cedar
import (
"bufio"
"encoding/gob"
"encoding/json"
"io"
"os"
)
// Save saves the cedar to an io.Writer,
// where dataType is either "json" or "gob".
func (da *Cedar) Save(out io.Writer, dataType string) error {
switch dataType {
case "gob", "GOB":
dataEecoder := gob.NewEncoder(out)
return dataEecoder.Encode(da.cedar)
case "json", "JSON":
dataEecoder := json.NewEncoder(out)
return dataEecoder.Encode(da.cedar)
}
return ErrInvalidDataType
}
// SaveToFile saves the cedar to a file,
// where dataType is either "json" or "gob".
func (da *Cedar) SaveToFile(fileName string, dataType string) error {
file, err := os.OpenFile(fileName, os.O_CREATE|os.O_WRONLY, 0666)
if err != nil {
return err
}
defer file.Close()
out := bufio.NewWriter(file)
defer out.Flush()
da.Save(out, dataType)
return nil
}
// Load loads the cedar from an io.Writer,
// where dataType is either "json" or "gob".
func (da *Cedar) Load(in io.Reader, dataType string) error {
switch dataType {
case "gob", "GOB":
dataDecoder := gob.NewDecoder(in)
return dataDecoder.Decode(da.cedar)
case "json", "JSON":
dataDecoder := json.NewDecoder(in)
return dataDecoder.Decode(da.cedar)
}
return ErrInvalidDataType
}
// LoadFromFile loads the cedar from a file,
// where dataType is either "json" or "gob".
func (da *Cedar) LoadFromFile(fileName string, dataType string) error {
file, err := os.OpenFile(fileName, os.O_RDONLY, 0600)
defer file.Close()
if err != nil {
return err
}
in := bufio.NewReader(file)
return da.Load(in, dataType)
}

4
vendor/github.com/boltdb/bolt/.gitignore generated vendored
View File

@ -1,4 +0,0 @@
*.prof
*.test
*.swp
/bin/

20
vendor/github.com/boltdb/bolt/LICENSE generated vendored
View File

@ -1,20 +0,0 @@
The MIT License (MIT)
Copyright (c) 2013 Ben Johnson
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

18
vendor/github.com/boltdb/bolt/Makefile generated vendored
View File

@ -1,18 +0,0 @@
BRANCH=`git rev-parse --abbrev-ref HEAD`
COMMIT=`git rev-parse --short HEAD`
GOLDFLAGS="-X main.branch $(BRANCH) -X main.commit $(COMMIT)"
default: build
race:
@go test -v -race -test.run="TestSimulate_(100op|1000op)"
# go get github.com/kisielk/errcheck
errcheck:
@errcheck -ignorepkg=bytes -ignore=os:Remove github.com/boltdb/bolt
test:
@go test -v -cover .
@go test -v ./cmd/bolt
.PHONY: fmt test

915
vendor/github.com/boltdb/bolt/README.md generated vendored
View File

@ -1,915 +0,0 @@
Bolt [![Coverage Status](https://coveralls.io/repos/boltdb/bolt/badge.svg?branch=master)](https://coveralls.io/r/boltdb/bolt?branch=master) [![GoDoc](https://godoc.org/github.com/boltdb/bolt?status.svg)](https://godoc.org/github.com/boltdb/bolt) ![Version](https://img.shields.io/badge/version-1.2.1-green.svg)
====
Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas]
[LMDB project][lmdb]. The goal of the project is to provide a simple,
fast, and reliable database for projects that don't require a full database
server such as Postgres or MySQL.
Since Bolt is meant to be used as such a low-level piece of functionality,
simplicity is key. The API will be small and only focus on getting values
and setting values. That's it.
[hyc_symas]: https://twitter.com/hyc_symas
[lmdb]: http://symas.com/mdb/
## Project Status
Bolt is stable, the API is fixed, and the file format is fixed. Full unit
test coverage and randomized black box testing are used to ensure database
consistency and thread safety. Bolt is currently used in high-load production
environments serving databases as large as 1TB. Many companies such as
Shopify and Heroku use Bolt-backed services every day.
## Table of Contents
- [Getting Started](#getting-started)
- [Installing](#installing)
- [Opening a database](#opening-a-database)
- [Transactions](#transactions)
- [Read-write transactions](#read-write-transactions)
- [Read-only transactions](#read-only-transactions)
- [Batch read-write transactions](#batch-read-write-transactions)
- [Managing transactions manually](#managing-transactions-manually)
- [Using buckets](#using-buckets)
- [Using key/value pairs](#using-keyvalue-pairs)
- [Autoincrementing integer for the bucket](#autoincrementing-integer-for-the-bucket)
- [Iterating over keys](#iterating-over-keys)
- [Prefix scans](#prefix-scans)
- [Range scans](#range-scans)
- [ForEach()](#foreach)
- [Nested buckets](#nested-buckets)
- [Database backups](#database-backups)
- [Statistics](#statistics)
- [Read-Only Mode](#read-only-mode)
- [Mobile Use (iOS/Android)](#mobile-use-iosandroid)
- [Resources](#resources)
- [Comparison with other databases](#comparison-with-other-databases)
- [Postgres, MySQL, & other relational databases](#postgres-mysql--other-relational-databases)
- [LevelDB, RocksDB](#leveldb-rocksdb)
- [LMDB](#lmdb)
- [Caveats & Limitations](#caveats--limitations)
- [Reading the Source](#reading-the-source)
- [Other Projects Using Bolt](#other-projects-using-bolt)
## Getting Started
### Installing
To start using Bolt, install Go and run `go get`:
```sh
$ go get github.com/boltdb/bolt/...
```
This will retrieve the library and install the `bolt` command line utility into
your `$GOBIN` path.
### Opening a database
The top-level object in Bolt is a `DB`. It is represented as a single file on
your disk and represents a consistent snapshot of your data.
To open your database, simply use the `bolt.Open()` function:
```go
package main
import (
"log"
"github.com/boltdb/bolt"
)
func main() {
// Open the my.db data file in your current directory.
// It will be created if it doesn't exist.
db, err := bolt.Open("my.db", 0600, nil)
if err != nil {
log.Fatal(err)
}
defer db.Close()
...
}
```
Please note that Bolt obtains a file lock on the data file so multiple processes
cannot open the same database at the same time. Opening an already open Bolt
database will cause it to hang until the other process closes it. To prevent
an indefinite wait you can pass a timeout option to the `Open()` function:
```go
db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})
```
### Transactions
Bolt allows only one read-write transaction at a time but allows as many
read-only transactions as you want at a time. Each transaction has a consistent
view of the data as it existed when the transaction started.
Individual transactions and all objects created from them (e.g. buckets, keys)
are not thread safe. To work with data in multiple goroutines you must start
a transaction for each one or use locking to ensure only one goroutine accesses
a transaction at a time. Creating transaction from the `DB` is thread safe.
Read-only transactions and read-write transactions should not depend on one
another and generally shouldn't be opened simultaneously in the same goroutine.
This can cause a deadlock as the read-write transaction needs to periodically
re-map the data file but it cannot do so while a read-only transaction is open.
#### Read-write transactions
To start a read-write transaction, you can use the `DB.Update()` function:
```go
err := db.Update(func(tx *bolt.Tx) error {
...
return nil
})
```
Inside the closure, you have a consistent view of the database. You commit the
transaction by returning `nil` at the end. You can also rollback the transaction
at any point by returning an error. All database operations are allowed inside
a read-write transaction.
Always check the return error as it will report any disk failures that can cause
your transaction to not complete. If you return an error within your closure
it will be passed through.
#### Read-only transactions
To start a read-only transaction, you can use the `DB.View()` function:
```go
err := db.View(func(tx *bolt.Tx) error {
...
return nil
})
```
You also get a consistent view of the database within this closure, however,
no mutating operations are allowed within a read-only transaction. You can only
retrieve buckets, retrieve values, and copy the database within a read-only
transaction.
#### Batch read-write transactions
Each `DB.Update()` waits for disk to commit the writes. This overhead
can be minimized by combining multiple updates with the `DB.Batch()`
function:
```go
err := db.Batch(func(tx *bolt.Tx) error {
...
return nil
})
```
Concurrent Batch calls are opportunistically combined into larger
transactions. Batch is only useful when there are multiple goroutines
calling it.
The trade-off is that `Batch` can call the given
function multiple times, if parts of the transaction fail. The
function must be idempotent and side effects must take effect only
after a successful return from `DB.Batch()`.
For example: don't display messages from inside the function, instead
set variables in the enclosing scope:
```go
var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
// Find last key in bucket, decode as bigendian uint64, increment
// by one, encode back to []byte, and add new key.
...
id = newValue
return nil
})
if err != nil {
return ...
}
fmt.Println("Allocated ID %d", id)
```
#### Managing transactions manually
The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()`
function. These helper functions will start the transaction, execute a function,
and then safely close your transaction if an error is returned. This is the
recommended way to use Bolt transactions.
However, sometimes you may want to manually start and end your transactions.
You can use the `DB.Begin()` function directly but **please** be sure to close
the transaction.
```go
// Start a writable transaction.
tx, err := db.Begin(true)
if err != nil {
return err
}
defer tx.Rollback()
// Use the transaction...
_, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
return err
}
// Commit the transaction and check for error.
if err := tx.Commit(); err != nil {
return err
}
```
The first argument to `DB.Begin()` is a boolean stating if the transaction
should be writable.
### Using buckets
Buckets are collections of key/value pairs within the database. All keys in a
bucket must be unique. You can create a bucket using the `DB.CreateBucket()`
function:
```go
db.Update(func(tx *bolt.Tx) error {
b, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
return fmt.Errorf("create bucket: %s", err)
}
return nil
})
```
You can also create a bucket only if it doesn't exist by using the
`Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this
function for all your top-level buckets after you open your database so you can
guarantee that they exist for future transactions.
To delete a bucket, simply call the `Tx.DeleteBucket()` function.
### Using key/value pairs
To save a key/value pair to a bucket, use the `Bucket.Put()` function:
```go
db.Update(func(tx *bolt.Tx) error {
b := tx.Bucket([]byte("MyBucket"))
err := b.Put([]byte("answer"), []byte("42"))
return err
})
```
This will set the value of the `"answer"` key to `"42"` in the `MyBucket`
bucket. To retrieve this value, we can use the `Bucket.Get()` function:
```go
db.View(func(tx *bolt.Tx) error {
b := tx.Bucket([]byte("MyBucket"))
v := b.Get([]byte("answer"))
fmt.Printf("The answer is: %s\n", v)
return nil
})
```
The `Get()` function does not return an error because its operation is
guaranteed to work (unless there is some kind of system failure). If the key
exists then it will return its byte slice value. If it doesn't exist then it
will return `nil`. It's important to note that you can have a zero-length value
set to a key which is different than the key not existing.
Use the `Bucket.Delete()` function to delete a key from the bucket.
Please note that values returned from `Get()` are only valid while the
transaction is open. If you need to use a value outside of the transaction
then you must use `copy()` to copy it to another byte slice.
### Autoincrementing integer for the bucket
By using the `NextSequence()` function, you can let Bolt determine a sequence
which can be used as the unique identifier for your key/value pairs. See the
example below.
```go
// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
func (s *Store) CreateUser(u *User) error {
return s.db.Update(func(tx *bolt.Tx) error {
// Retrieve the users bucket.
// This should be created when the DB is first opened.
b := tx.Bucket([]byte("users"))
// Generate ID for the user.
// This returns an error only if the Tx is closed or not writeable.
// That can't happen in an Update() call so I ignore the error check.
id, _ := b.NextSequence()
u.ID = int(id)
// Marshal user data into bytes.
buf, err := json.Marshal(u)
if err != nil {
return err
}
// Persist bytes to users bucket.
return b.Put(itob(u.ID), buf)
})
}
// itob returns an 8-byte big endian representation of v.
func itob(v int) []byte {
b := make([]byte, 8)
binary.BigEndian.PutUint64(b, uint64(v))
return b
}
type User struct {
ID int
...
}
```
### Iterating over keys
Bolt stores its keys in byte-sorted order within a bucket. This makes sequential
iteration over these keys extremely fast. To iterate over keys we'll use a
`Cursor`:
```go
db.View(func(tx *bolt.Tx) error {
// Assume bucket exists and has keys
b := tx.Bucket([]byte("MyBucket"))
c := b.Cursor()
for k, v := c.First(); k != nil; k, v = c.Next() {
fmt.Printf("key=%s, value=%s\n", k, v)
}
return nil
})
```
The cursor allows you to move to a specific point in the list of keys and move
forward or backward through the keys one at a time.
The following functions are available on the cursor:
```
First() Move to the first key.
Last() Move to the last key.
Seek() Move to a specific key.
Next() Move to the next key.
Prev() Move to the previous key.
```
Each of those functions has a return signature of `(key []byte, value []byte)`.
When you have iterated to the end of the cursor then `Next()` will return a
`nil` key. You must seek to a position using `First()`, `Last()`, or `Seek()`
before calling `Next()` or `Prev()`. If you do not seek to a position then
these functions will return a `nil` key.
During iteration, if the key is non-`nil` but the value is `nil`, that means
the key refers to a bucket rather than a value. Use `Bucket.Bucket()` to
access the sub-bucket.
#### Prefix scans
To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`:
```go
db.View(func(tx *bolt.Tx) error {
// Assume bucket exists and has keys
c := tx.Bucket([]byte("MyBucket")).Cursor()
prefix := []byte("1234")
for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
fmt.Printf("key=%s, value=%s\n", k, v)
}
return nil
})
```
#### Range scans
Another common use case is scanning over a range such as a time range. If you
use a sortable time encoding such as RFC3339 then you can query a specific
date range like this:
```go
db.View(func(tx *bolt.Tx) error {
// Assume our events bucket exists and has RFC3339 encoded time keys.
c := tx.Bucket([]byte("Events")).Cursor()
// Our time range spans the 90's decade.
min := []byte("1990-01-01T00:00:00Z")
max := []byte("2000-01-01T00:00:00Z")
// Iterate over the 90's.
for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
fmt.Printf("%s: %s\n", k, v)
}
return nil
})
```
Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.
#### ForEach()
You can also use the function `ForEach()` if you know you'll be iterating over
all the keys in a bucket:
```go
db.View(func(tx *bolt.Tx) error {
// Assume bucket exists and has keys
b := tx.Bucket([]byte("MyBucket"))
b.ForEach(func(k, v []byte) error {
fmt.Printf("key=%s, value=%s\n", k, v)
return nil
})
return nil
})
```
Please note that keys and values in `ForEach()` are only valid while
the transaction is open. If you need to use a key or value outside of
the transaction, you must use `copy()` to copy it to another byte
slice.
### Nested buckets
You can also store a bucket in a key to create nested buckets. The API is the
same as the bucket management API on the `DB` object:
```go
func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
func (*Bucket) DeleteBucket(key []byte) error
```
Say you had a multi-tenant application where the root level bucket was the account bucket. Inside of this bucket was a sequence of accounts which themselves are buckets. And inside the sequence bucket you could have many buckets pertaining to the Account itself (Users, Notes, etc) isolating the information into logical groupings.
```go
// createUser creates a new user in the given account.
func createUser(accountID int, u *User) error {
// Start the transaction.
tx, err := db.Begin(true)
if err != nil {
return err
}
defer tx.Rollback()
// Retrieve the root bucket for the account.
// Assume this has already been created when the account was set up.
root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10)))
// Setup the users bucket.
bkt, err := root.CreateBucketIfNotExists([]byte("USERS"))
if err != nil {
return err
}
// Generate an ID for the new user.
userID, err := bkt.NextSequence()
if err != nil {
return err
}
u.ID = userID
// Marshal and save the encoded user.
if buf, err := json.Marshal(u); err != nil {
return err
} else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil {
return err
}
// Commit the transaction.
if err := tx.Commit(); err != nil {
return err
}
return nil
}
```
### Database backups
Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()`
function to write a consistent view of the database to a writer. If you call
this from a read-only transaction, it will perform a hot backup and not block
your other database reads and writes.
By default, it will use a regular file handle which will utilize the operating
system's page cache. See the [`Tx`](https://godoc.org/github.com/boltdb/bolt#Tx)
documentation for information about optimizing for larger-than-RAM datasets.
One common use case is to backup over HTTP so you can use tools like `cURL` to
do database backups:
```go
func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
err := db.View(func(tx *bolt.Tx) error {
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
_, err := tx.WriteTo(w)
return err
})
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
```
Then you can backup using this command:
```sh
$ curl http://localhost/backup > my.db
```
Or you can open your browser to `http://localhost/backup` and it will download
automatically.
If you want to backup to another file you can use the `Tx.CopyFile()` helper
function.
### Statistics
The database keeps a running count of many of the internal operations it
performs so you can better understand what's going on. By grabbing a snapshot
of these stats at two points in time we can see what operations were performed
in that time range.
For example, we could start a goroutine to log stats every 10 seconds:
```go
go func() {
// Grab the initial stats.
prev := db.Stats()
for {
// Wait for 10s.
time.Sleep(10 * time.Second)
// Grab the current stats and diff them.
stats := db.Stats()
diff := stats.Sub(&prev)
// Encode stats to JSON and print to STDERR.
json.NewEncoder(os.Stderr).Encode(diff)
// Save stats for the next loop.
prev = stats
}
}()
```
It's also useful to pipe these stats to a service such as statsd for monitoring
or to provide an HTTP endpoint that will perform a fixed-length sample.
### Read-Only Mode
Sometimes it is useful to create a shared, read-only Bolt database. To this,
set the `Options.ReadOnly` flag when opening your database. Read-only mode
uses a shared lock to allow multiple processes to read from the database but
it will block any processes from opening the database in read-write mode.
```go
db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
if err != nil {
log.Fatal(err)
}
```
### Mobile Use (iOS/Android)
Bolt is able to run on mobile devices by leveraging the binding feature of the
[gomobile](https://github.com/golang/mobile) tool. Create a struct that will
contain your database logic and a reference to a `*bolt.DB` with a initializing
constructor that takes in a filepath where the database file will be stored.
Neither Android nor iOS require extra permissions or cleanup from using this method.
```go
func NewBoltDB(filepath string) *BoltDB {
db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
if err != nil {
log.Fatal(err)
}
return &BoltDB{db}
}
type BoltDB struct {
db *bolt.DB
...
}
func (b *BoltDB) Path() string {
return b.db.Path()
}
func (b *BoltDB) Close() {
b.db.Close()
}
```
Database logic should be defined as methods on this wrapper struct.
To initialize this struct from the native language (both platforms now sync
their local storage to the cloud. These snippets disable that functionality for the
database file):
#### Android
```java
String path;
if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
path = getNoBackupFilesDir().getAbsolutePath();
} else{
path = getFilesDir().getAbsolutePath();
}
Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)
```
#### iOS
```objc
- (void)demo {
NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
NSUserDomainMask,
YES) objectAtIndex:0];
GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
[self addSkipBackupAttributeToItemAtPath:demo.path];
//Some DB Logic would go here
[demo close];
}
- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
{
NSURL* URL= [NSURL fileURLWithPath: filePathString];
assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);
NSError *error = nil;
BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
forKey: NSURLIsExcludedFromBackupKey error: &error];
if(!success){
NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
}
return success;
}
```
## Resources
For more information on getting started with Bolt, check out the following articles:
* [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch).
* [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville
## Comparison with other databases
### Postgres, MySQL, & other relational databases
Relational databases structure data into rows and are only accessible through
the use of SQL. This approach provides flexibility in how you store and query
your data but also incurs overhead in parsing and planning SQL statements. Bolt
accesses all data by a byte slice key. This makes Bolt fast to read and write
data by key but provides no built-in support for joining values together.
Most relational databases (with the exception of SQLite) are standalone servers
that run separately from your application. This gives your systems
flexibility to connect multiple application servers to a single database
server but also adds overhead in serializing and transporting data over the
network. Bolt runs as a library included in your application so all data access
has to go through your application's process. This brings data closer to your
application but limits multi-process access to the data.
### LevelDB, RocksDB
LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that
they are libraries bundled into the application, however, their underlying
structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes
random writes by using a write ahead log and multi-tiered, sorted files called
SSTables. Bolt uses a B+tree internally and only a single file. Both approaches
have trade-offs.
If you require a high random write throughput (>10,000 w/sec) or you need to use
spinning disks then LevelDB could be a good choice. If your application is
read-heavy or does a lot of range scans then Bolt could be a good choice.
One other important consideration is that LevelDB does not have transactions.
It supports batch writing of key/values pairs and it supports read snapshots
but it will not give you the ability to do a compare-and-swap operation safely.
Bolt supports fully serializable ACID transactions.
### LMDB
Bolt was originally a port of LMDB so it is architecturally similar. Both use
a B+tree, have ACID semantics with fully serializable transactions, and support
lock-free MVCC using a single writer and multiple readers.
The two projects have somewhat diverged. LMDB heavily focuses on raw performance
while Bolt has focused on simplicity and ease of use. For example, LMDB allows
several unsafe actions such as direct writes for the sake of performance. Bolt
opts to disallow actions which can leave the database in a corrupted state. The
only exception to this in Bolt is `DB.NoSync`.
There are also a few differences in API. LMDB requires a maximum mmap size when
opening an `mdb_env` whereas Bolt will handle incremental mmap resizing
automatically. LMDB overloads the getter and setter functions with multiple
flags whereas Bolt splits these specialized cases into their own functions.
## Caveats & Limitations
It's important to pick the right tool for the job and Bolt is no exception.
Here are a few things to note when evaluating and using Bolt:
* Bolt is good for read intensive workloads. Sequential write performance is
also fast but random writes can be slow. You can use `DB.Batch()` or add a
write-ahead log to help mitigate this issue.
* Bolt uses a B+tree internally so there can be a lot of random page access.
SSDs provide a significant performance boost over spinning disks.
* Try to avoid long running read transactions. Bolt uses copy-on-write so
old pages cannot be reclaimed while an old transaction is using them.
* Byte slices returned from Bolt are only valid during a transaction. Once the
transaction has been committed or rolled back then the memory they point to
can be reused by a new page or can be unmapped from virtual memory and you'll
see an `unexpected fault address` panic when accessing it.
* Bolt uses an exclusive write lock on the database file so it cannot be
shared by multiple processes.
* Be careful when using `Bucket.FillPercent`. Setting a high fill percent for
buckets that have random inserts will cause your database to have very poor
page utilization.
* Use larger buckets in general. Smaller buckets causes poor page utilization
once they become larger than the page size (typically 4KB).
* Bulk loading a lot of random writes into a new bucket can be slow as the
page will not split until the transaction is committed. Randomly inserting
more than 100,000 key/value pairs into a single new bucket in a single
transaction is not advised.
* Bolt uses a memory-mapped file so the underlying operating system handles the
caching of the data. Typically, the OS will cache as much of the file as it
can in memory and will release memory as needed to other processes. This means
that Bolt can show very high memory usage when working with large databases.
However, this is expected and the OS will release memory as needed. Bolt can
handle databases much larger than the available physical RAM, provided its
memory-map fits in the process virtual address space. It may be problematic
on 32-bits systems.
* The data structures in the Bolt database are memory mapped so the data file
will be endian specific. This means that you cannot copy a Bolt file from a
little endian machine to a big endian machine and have it work. For most
users this is not a concern since most modern CPUs are little endian.
* Because of the way pages are laid out on disk, Bolt cannot truncate data files
and return free pages back to the disk. Instead, Bolt maintains a free list
of unused pages within its data file. These free pages can be reused by later
transactions. This works well for many use cases as databases generally tend
to grow. However, it's important to note that deleting large chunks of data
will not allow you to reclaim that space on disk.
For more information on page allocation, [see this comment][page-allocation].
[page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638
## Reading the Source
Bolt is a relatively small code base (<3KLOC) for an embedded, serializable,
transactional key/value database so it can be a good starting point for people
interested in how databases work.
The best places to start are the main entry points into Bolt:
- `Open()` - Initializes the reference to the database. It's responsible for
creating the database if it doesn't exist, obtaining an exclusive lock on the
file, reading the meta pages, & memory-mapping the file.
- `DB.Begin()` - Starts a read-only or read-write transaction depending on the
value of the `writable` argument. This requires briefly obtaining the "meta"
lock to keep track of open transactions. Only one read-write transaction can
exist at a time so the "rwlock" is acquired during the life of a read-write
transaction.
- `Bucket.Put()` - Writes a key/value pair into a bucket. After validating the
arguments, a cursor is used to traverse the B+tree to the page and position
where they key & value will be written. Once the position is found, the bucket
materializes the underlying page and the page's parent pages into memory as
"nodes". These nodes are where mutations occur during read-write transactions.
These changes get flushed to disk during commit.
- `Bucket.Get()` - Retrieves a key/value pair from a bucket. This uses a cursor
to move to the page & position of a key/value pair. During a read-only
transaction, the key and value data is returned as a direct reference to the
underlying mmap file so there's no allocation overhead. For read-write
transactions, this data may reference the mmap file or one of the in-memory
node values.
- `Cursor` - This object is simply for traversing the B+tree of on-disk pages
or in-memory nodes. It can seek to a specific key, move to the first or last
value, or it can move forward or backward. The cursor handles the movement up
and down the B+tree transparently to the end user.
- `Tx.Commit()` - Converts the in-memory dirty nodes and the list of free pages
into pages to be written to disk. Writing to disk then occurs in two phases.
First, the dirty pages are written to disk and an `fsync()` occurs. Second, a
new meta page with an incremented transaction ID is written and another
`fsync()` occurs. This two phase write ensures that partially written data
pages are ignored in the event of a crash since the meta page pointing to them
is never written. Partially written meta pages are invalidated because they
are written with a checksum.
If you have additional notes that could be helpful for others, please submit
them via pull request.
## Other Projects Using Bolt
Below is a list of public, open source projects that use Bolt:
* [BoltDbWeb](https://github.com/evnix/boltdbweb) - A web based GUI for BoltDB files.
* [Operation Go: A Routine Mission](http://gocode.io) - An online programming game for Golang using Bolt for user accounts and a leaderboard.
* [Bazil](https://bazil.org/) - A file system that lets your data reside where it is most convenient for it to reside.
* [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
* [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics.
* [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects.
* [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday.
* [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations.
* [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite.
* [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
* [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka.
* [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed.
* [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt.
* [photosite/session](https://godoc.org/bitbucket.org/kardianos/photosite/session) - Sessions for a photo viewing site.
* [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage.
* [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters.
* [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend.
* [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
* [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
* [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read.
* [InfluxDB](https://influxdata.com) - Scalable datastore for metrics, events, and real-time analytics.
* [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data.
* [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system.
* [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
* [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
* [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems.
* [stow](https://github.com/djherbis/stow) - a persistence manager for objects
backed by boltdb.
* [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining
simple tx and key scans.
* [mbuckets](https://github.com/abhigupta912/mbuckets) - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
* [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service
* [Go Report Card](https://goreportcard.com/) - Go code quality report cards as a (free and open source) service.
* [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
* [lru](https://github.com/crowdriff/lru) - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
* [Storm](https://github.com/asdine/storm) - Simple and powerful ORM for BoltDB.
* [GoWebApp](https://github.com/josephspurrier/gowebapp) - A basic MVC web application in Go using BoltDB.
* [SimpleBolt](https://github.com/xyproto/simplebolt) - A simple way to use BoltDB. Deals mainly with strings.
* [Algernon](https://github.com/xyproto/algernon) - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend.
* [MuLiFS](https://github.com/dankomiocevic/mulifs) - Music Library Filesystem creates a filesystem to organise your music files.
* [GoShort](https://github.com/pankajkhairnar/goShort) - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter.
* [torrent](https://github.com/anacrolix/torrent) - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development.
* [gopherpit](https://github.com/gopherpit/gopherpit) - A web service to manage Go remote import paths with custom domains
* [bolter](https://github.com/hasit/bolter) - Command-line app for viewing BoltDB file in your terminal.
* [btcwallet](https://github.com/btcsuite/btcwallet) - A bitcoin wallet.
* [dcrwallet](https://github.com/decred/dcrwallet) - A wallet for the Decred cryptocurrency.
* [Ironsmith](https://github.com/timshannon/ironsmith) - A simple, script-driven continuous integration (build - > test -> release) tool, with no external dependencies
* [BoltHold](https://github.com/timshannon/bolthold) - An embeddable NoSQL store for Go types built on BoltDB
If you are using Bolt in a project please send a pull request to add it to the list.

18
vendor/github.com/boltdb/bolt/appveyor.yml generated vendored
View File

@ -1,18 +0,0 @@
version: "{build}"
os: Windows Server 2012 R2
clone_folder: c:\gopath\src\github.com\boltdb\bolt
environment:
GOPATH: c:\gopath
install:
- echo %PATH%
- echo %GOPATH%
- go version
- go env
- go get -v -t ./...
build_script:
- go test -v ./...

10
vendor/github.com/boltdb/bolt/bolt_386.go generated vendored
View File

@ -1,10 +0,0 @@
package bolt
// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0x7FFFFFFF // 2GB
// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0xFFFFFFF
// Are unaligned load/stores broken on this arch?
var brokenUnaligned = false

10
vendor/github.com/boltdb/bolt/bolt_amd64.go generated vendored
View File

@ -1,10 +0,0 @@
package bolt
// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0xFFFFFFFFFFFF // 256TB
// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0x7FFFFFFF
// Are unaligned load/stores broken on this arch?
var brokenUnaligned = false

28
vendor/github.com/boltdb/bolt/bolt_arm.go generated vendored
View File

@ -1,28 +0,0 @@
package bolt
import "unsafe"
// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0x7FFFFFFF // 2GB
// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0xFFFFFFF
// Are unaligned load/stores broken on this arch?
var brokenUnaligned bool
func init() {
// Simple check to see whether this arch handles unaligned load/stores
// correctly.
// ARM9 and older devices require load/stores to be from/to aligned
// addresses. If not, the lower 2 bits are cleared and that address is
// read in a jumbled up order.
// See http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka15414.html
raw := [6]byte{0xfe, 0xef, 0x11, 0x22, 0x22, 0x11}
val := *(*uint32)(unsafe.Pointer(uintptr(unsafe.Pointer(&raw)) + 2))
brokenUnaligned = val != 0x11222211
}

12
vendor/github.com/boltdb/bolt/bolt_arm64.go generated vendored
View File

@ -1,12 +0,0 @@
// +build arm64
package bolt
// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0xFFFFFFFFFFFF // 256TB
// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0x7FFFFFFF
// Are unaligned load/stores broken on this arch?
var brokenUnaligned = false

10
vendor/github.com/boltdb/bolt/bolt_linux.go generated vendored
View File

@ -1,10 +0,0 @@
package bolt
import (
"syscall"
)
// fdatasync flushes written data to a file descriptor.
func fdatasync(db *DB) error {
return syscall.Fdatasync(int(db.file.Fd()))
}

27
vendor/github.com/boltdb/bolt/bolt_openbsd.go generated vendored
View File

@ -1,27 +0,0 @@
package bolt
import (
"syscall"
"unsafe"
)
const (
msAsync = 1 << iota // perform asynchronous writes
msSync // perform synchronous writes
msInvalidate // invalidate cached data
)
func msync(db *DB) error {
_, _, errno := syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(db.data)), uintptr(db.datasz), msInvalidate)
if errno != 0 {
return errno
}
return nil
}
func fdatasync(db *DB) error {
if db.data != nil {
return msync(db)
}
return db.file.Sync()
}

9
vendor/github.com/boltdb/bolt/bolt_ppc.go generated vendored
View File

@ -1,9 +0,0 @@
// +build ppc
package bolt
// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0x7FFFFFFF // 2GB
// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0xFFFFFFF

12
vendor/github.com/boltdb/bolt/bolt_ppc64.go generated vendored
View File

@ -1,12 +0,0 @@
// +build ppc64
package bolt
// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0xFFFFFFFFFFFF // 256TB
// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0x7FFFFFFF
// Are unaligned load/stores broken on this arch?
var brokenUnaligned = false

12
vendor/github.com/boltdb/bolt/bolt_ppc64le.go generated vendored
View File

@ -1,12 +0,0 @@
// +build ppc64le
package bolt
// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0xFFFFFFFFFFFF // 256TB
// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0x7FFFFFFF
// Are unaligned load/stores broken on this arch?
var brokenUnaligned = false

12
vendor/github.com/boltdb/bolt/bolt_s390x.go generated vendored
View File

@ -1,12 +0,0 @@
// +build s390x
package bolt
// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0xFFFFFFFFFFFF // 256TB
// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0x7FFFFFFF
// Are unaligned load/stores broken on this arch?
var brokenUnaligned = false

89
vendor/github.com/boltdb/bolt/bolt_unix.go generated vendored
View File

@ -1,89 +0,0 @@
// +build !windows,!plan9,!solaris
package bolt
import (
"fmt"
"os"
"syscall"
"time"
"unsafe"
)
// flock acquires an advisory lock on a file descriptor.
func flock(db *DB, mode os.FileMode, exclusive bool, timeout time.Duration) error {
var t time.Time
for {
// If we're beyond our timeout then return an error.
// This can only occur after we've attempted a flock once.
if t.IsZero() {
t = time.Now()
} else if timeout > 0 && time.Since(t) > timeout {
return ErrTimeout
}
flag := syscall.LOCK_SH
if exclusive {
flag = syscall.LOCK_EX
}
// Otherwise attempt to obtain an exclusive lock.
err := syscall.Flock(int(db.file.Fd()), flag|syscall.LOCK_NB)
if err == nil {
return nil
} else if err != syscall.EWOULDBLOCK {
return err
}
// Wait for a bit and try again.
time.Sleep(50 * time.Millisecond)
}
}
// funlock releases an advisory lock on a file descriptor.
func funlock(db *DB) error {
return syscall.Flock(int(db.file.Fd()), syscall.LOCK_UN)
}
// mmap memory maps a DB's data file.
func mmap(db *DB, sz int) error {
// Map the data file to memory.
b, err := syscall.Mmap(int(db.file.Fd()), 0, sz, syscall.PROT_READ, syscall.MAP_SHARED|db.MmapFlags)
if err != nil {
return err
}
// Advise the kernel that the mmap is accessed randomly.
if err := madvise(b, syscall.MADV_RANDOM); err != nil {
return fmt.Errorf("madvise: %s", err)
}
// Save the original byte slice and convert to a byte array pointer.
db.dataref = b
db.data = (*[maxMapSize]byte)(unsafe.Pointer(&b[0]))
db.datasz = sz
return nil
}
// munmap unmaps a DB's data file from memory.
func munmap(db *DB) error {
// Ignore the unmap if we have no mapped data.
if db.dataref == nil {
return nil
}
// Unmap using the original byte slice.
err := syscall.Munmap(db.dataref)
db.dataref = nil
db.data = nil
db.datasz = 0
return err
}
// NOTE: This function is copied from stdlib because it is not available on darwin.
func madvise(b []byte, advice int) (err error) {
_, _, e1 := syscall.Syscall(syscall.SYS_MADVISE, uintptr(unsafe.Pointer(&b[0])), uintptr(len(b)), uintptr(advice))
if e1 != 0 {
err = e1
}
return
}

90
vendor/github.com/boltdb/bolt/bolt_unix_solaris.go generated vendored
View File

@ -1,90 +0,0 @@
package bolt
import (
"fmt"
"os"
"syscall"
"time"
"unsafe"
"golang.org/x/sys/unix"
)
// flock acquires an advisory lock on a file descriptor.
func flock(db *DB, mode os.FileMode, exclusive bool, timeout time.Duration) error {
var t time.Time
for {
// If we're beyond our timeout then return an error.
// This can only occur after we've attempted a flock once.
if t.IsZero() {
t = time.Now()
} else if timeout > 0 && time.Since(t) > timeout {
return ErrTimeout
}
var lock syscall.Flock_t
lock.Start = 0
lock.Len = 0
lock.Pid = 0
lock.Whence = 0
lock.Pid = 0
if exclusive {
lock.Type = syscall.F_WRLCK
} else {
lock.Type = syscall.F_RDLCK
}
err := syscall.FcntlFlock(db.file.Fd(), syscall.F_SETLK, &lock)
if err == nil {
return nil
} else if err != syscall.EAGAIN {
return err
}
// Wait for a bit and try again.
time.Sleep(50 * time.Millisecond)
}
}
// funlock releases an advisory lock on a file descriptor.
func funlock(db *DB) error {
var lock syscall.Flock_t
lock.Start = 0
lock.Len = 0
lock.Type = syscall.F_UNLCK
lock.Whence = 0
return syscall.FcntlFlock(uintptr(db.file.Fd()), syscall.F_SETLK, &lock)
}
// mmap memory maps a DB's data file.
func mmap(db *DB, sz int) error {
// Map the data file to memory.
b, err := unix.Mmap(int(db.file.Fd()), 0, sz, syscall.PROT_READ, syscall.MAP_SHARED|db.MmapFlags)
if err != nil {
return err
}
// Advise the kernel that the mmap is accessed randomly.
if err := unix.Madvise(b, syscall.MADV_RANDOM); err != nil {
return fmt.Errorf("madvise: %s", err)
}
// Save the original byte slice and convert to a byte array pointer.
db.dataref = b
db.data = (*[maxMapSize]byte)(unsafe.Pointer(&b[0]))
db.datasz = sz
return nil
}
// munmap unmaps a DB's data file from memory.
func munmap(db *DB) error {
// Ignore the unmap if we have no mapped data.
if db.dataref == nil {
return nil
}
// Unmap using the original byte slice.
err := unix.Munmap(db.dataref)
db.dataref = nil
db.data = nil
db.datasz = 0
return err
}

144
vendor/github.com/boltdb/bolt/bolt_windows.go generated vendored
View File

@ -1,144 +0,0 @@
package bolt
import (
"fmt"
"os"
"syscall"
"time"
"unsafe"
)
// LockFileEx code derived from golang build filemutex_windows.go @ v1.5.1
var (
modkernel32 = syscall.NewLazyDLL("kernel32.dll")
procLockFileEx = modkernel32.NewProc("LockFileEx")
procUnlockFileEx = modkernel32.NewProc("UnlockFileEx")
)
const (
lockExt = ".lock"
// see https://msdn.microsoft.com/en-us/library/windows/desktop/aa365203(v=vs.85).aspx
flagLockExclusive = 2
flagLockFailImmediately = 1
// see https://msdn.microsoft.com/en-us/library/windows/desktop/ms681382(v=vs.85).aspx
errLockViolation syscall.Errno = 0x21
)
func lockFileEx(h syscall.Handle, flags, reserved, locklow, lockhigh uint32, ol *syscall.Overlapped) (err error) {
r, _, err := procLockFileEx.Call(uintptr(h), uintptr(flags), uintptr(reserved), uintptr(locklow), uintptr(lockhigh), uintptr(unsafe.Pointer(ol)))
if r == 0 {
return err
}
return nil
}
func unlockFileEx(h syscall.Handle, reserved, locklow, lockhigh uint32, ol *syscall.Overlapped) (err error) {
r, _, err := procUnlockFileEx.Call(uintptr(h), uintptr(reserved), uintptr(locklow), uintptr(lockhigh), uintptr(unsafe.Pointer(ol)), 0)
if r == 0 {
return err
}
return nil
}
// fdatasync flushes written data to a file descriptor.
func fdatasync(db *DB) error {
return db.file.Sync()
}
// flock acquires an advisory lock on a file descriptor.
func flock(db *DB, mode os.FileMode, exclusive bool, timeout time.Duration) error {
// Create a separate lock file on windows because a process
// cannot share an exclusive lock on the same file. This is
// needed during Tx.WriteTo().
f, err := os.OpenFile(db.path+lockExt, os.O_CREATE, mode)
if err != nil {
return err
}
db.lockfile = f
var t time.Time
for {
// If we're beyond our timeout then return an error.
// This can only occur after we've attempted a flock once.
if t.IsZero() {
t = time.Now()
} else if timeout > 0 && time.Since(t) > timeout {
return ErrTimeout
}
var flag uint32 = flagLockFailImmediately
if exclusive {
flag |= flagLockExclusive
}
err := lockFileEx(syscall.Handle(db.lockfile.Fd()), flag, 0, 1, 0, &syscall.Overlapped{})
if err == nil {
return nil
} else if err != errLockViolation {
return err
}
// Wait for a bit and try again.
time.Sleep(50 * time.Millisecond)
}
}
// funlock releases an advisory lock on a file descriptor.
func funlock(db *DB) error {
err := unlockFileEx(syscall.Handle(db.lockfile.Fd()), 0, 1, 0, &syscall.Overlapped{})
db.lockfile.Close()
os.Remove(db.path + lockExt)
return err
}
// mmap memory maps a DB's data file.
// Based on: https://github.com/edsrzf/mmap-go
func mmap(db *DB, sz int) error {
if !db.readOnly {
// Truncate the database to the size of the mmap.
if err := db.file.Truncate(int64(sz)); err != nil {
return fmt.Errorf("truncate: %s", err)
}
}
// Open a file mapping handle.
sizelo := uint32(sz >> 32)
sizehi := uint32(sz) & 0xffffffff
h, errno := syscall.CreateFileMapping(syscall.Handle(db.file.Fd()), nil, syscall.PAGE_READONLY, sizelo, sizehi, nil)
if h == 0 {
return os.NewSyscallError("CreateFileMapping", errno)
}
// Create the memory map.
addr, errno := syscall.MapViewOfFile(h, syscall.FILE_MAP_READ, 0, 0, uintptr(sz))
if addr == 0 {
return os.NewSyscallError("MapViewOfFile", errno)
}
// Close mapping handle.
if err := syscall.CloseHandle(syscall.Handle(h)); err != nil {
return os.NewSyscallError("CloseHandle", err)
}
// Convert to a byte array.
db.data = ((*[maxMapSize]byte)(unsafe.Pointer(addr)))
db.datasz = sz
return nil
}
// munmap unmaps a pointer from a file.
// Based on: https://github.com/edsrzf/mmap-go
func munmap(db *DB) error {
if db.data == nil {
return nil
}
addr := (uintptr)(unsafe.Pointer(&db.data[0]))
if err := syscall.UnmapViewOfFile(addr); err != nil {
return os.NewSyscallError("UnmapViewOfFile", err)
}
return nil
}

8
vendor/github.com/boltdb/bolt/boltsync_unix.go generated vendored
View File

@ -1,8 +0,0 @@
// +build !windows,!plan9,!linux,!openbsd
package bolt
// fdatasync flushes written data to a file descriptor.
func fdatasync(db *DB) error {
return db.file.Sync()
}

777
vendor/github.com/boltdb/bolt/bucket.go generated vendored
View File

@ -1,777 +0,0 @@
package bolt
import (
"bytes"
"fmt"
"unsafe"
)
const (
// MaxKeySize is the maximum length of a key, in bytes.
MaxKeySize = 32768
// MaxValueSize is the maximum length of a value, in bytes.
MaxValueSize = (1 << 31) - 2
)
const (
maxUint = ^uint(0)
minUint = 0
maxInt = int(^uint(0) >> 1)
minInt = -maxInt - 1
)
const bucketHeaderSize = int(unsafe.Sizeof(bucket{}))
const (
minFillPercent = 0.1
maxFillPercent = 1.0
)
// DefaultFillPercent is the percentage that split pages are filled.
// This value can be changed by setting Bucket.FillPercent.
const DefaultFillPercent = 0.5
// Bucket represents a collection of key/value pairs inside the database.
type Bucket struct {
*bucket
tx *Tx // the associated transaction
buckets map[string]*Bucket // subbucket cache
page *page // inline page reference
rootNode *node // materialized node for the root page.
nodes map[pgid]*node // node cache
// Sets the threshold for filling nodes when they split. By default,
// the bucket will fill to 50% but it can be useful to increase this
// amount if you know that your write workloads are mostly append-only.
//
// This is non-persisted across transactions so it must be set in every Tx.
FillPercent float64
}
// bucket represents the on-file representation of a bucket.
// This is stored as the "value" of a bucket key. If the bucket is small enough,
// then its root page can be stored inline in the "value", after the bucket
// header. In the case of inline buckets, the "root" will be 0.
type bucket struct {
root pgid // page id of the bucket's root-level page
sequence uint64 // monotonically incrementing, used by NextSequence()
}
// newBucket returns a new bucket associated with a transaction.
func newBucket(tx *Tx) Bucket {
var b = Bucket{tx: tx, FillPercent: DefaultFillPercent}
if tx.writable {
b.buckets = make(map[string]*Bucket)
b.nodes = make(map[pgid]*node)
}
return b
}
// Tx returns the tx of the bucket.
func (b *Bucket) Tx() *Tx {
return b.tx
}
// Root returns the root of the bucket.
func (b *Bucket) Root() pgid {
return b.root
}
// Writable returns whether the bucket is writable.
func (b *Bucket) Writable() bool {
return b.tx.writable
}
// Cursor creates a cursor associated with the bucket.
// The cursor is only valid as long as the transaction is open.
// Do not use a cursor after the transaction is closed.
func (b *Bucket) Cursor() *Cursor {
// Update transaction statistics.
b.tx.stats.CursorCount++
// Allocate and return a cursor.
return &Cursor{
bucket: b,
stack: make([]elemRef, 0),
}
}
// Bucket retrieves a nested bucket by name.
// Returns nil if the bucket does not exist.
// The bucket instance is only valid for the lifetime of the transaction.
func (b *Bucket) Bucket(name []byte) *Bucket {
if b.buckets != nil {
if child := b.buckets[string(name)]; child != nil {
return child
}
}
// Move cursor to key.
c := b.Cursor()
k, v, flags := c.seek(name)
// Return nil if the key doesn't exist or it is not a bucket.
if !bytes.Equal(name, k) || (flags&bucketLeafFlag) == 0 {
return nil
}
// Otherwise create a bucket and cache it.
var child = b.openBucket(v)
if b.buckets != nil {
b.buckets[string(name)] = child
}
return child
}
// Helper method that re-interprets a sub-bucket value
// from a parent into a Bucket
func (b *Bucket) openBucket(value []byte) *Bucket {
var child = newBucket(b.tx)
// If unaligned load/stores are broken on this arch and value is
// unaligned simply clone to an aligned byte array.
unaligned := brokenUnaligned && uintptr(unsafe.Pointer(&value[0]))&3 != 0
if unaligned {
value = cloneBytes(value)
}
// If this is a writable transaction then we need to copy the bucket entry.
// Read-only transactions can point directly at the mmap entry.
if b.tx.writable && !unaligned {
child.bucket = &bucket{}
*child.bucket = *(*bucket)(unsafe.Pointer(&value[0]))
} else {
child.bucket = (*bucket)(unsafe.Pointer(&value[0]))
}
// Save a reference to the inline page if the bucket is inline.
if child.root == 0 {
child.page = (*page)(unsafe.Pointer(&value[bucketHeaderSize]))
}
return &child
}
// CreateBucket creates a new bucket at the given key and returns the new bucket.
// Returns an error if the key already exists, if the bucket name is blank, or if the bucket name is too long.
// The bucket instance is only valid for the lifetime of the transaction.
func (b *Bucket) CreateBucket(key []byte) (*Bucket, error) {
if b.tx.db == nil {
return nil, ErrTxClosed
} else if !b.tx.writable {
return nil, ErrTxNotWritable
} else if len(key) == 0 {
return nil, ErrBucketNameRequired
}
// Move cursor to correct position.
c := b.Cursor()
k, _, flags := c.seek(key)
// Return an error if there is an existing key.
if bytes.Equal(key, k) {
if (flags & bucketLeafFlag) != 0 {
return nil, ErrBucketExists
}
return nil, ErrIncompatibleValue
}
// Create empty, inline bucket.
var bucket = Bucket{
bucket: &bucket{},
rootNode: &node{isLeaf: true},
FillPercent: DefaultFillPercent,
}
var value = bucket.write()
// Insert into node.
key = cloneBytes(key)
c.node().put(key, key, value, 0, bucketLeafFlag)
// Since subbuckets are not allowed on inline buckets, we need to
// dereference the inline page, if it exists. This will cause the bucket
// to be treated as a regular, non-inline bucket for the rest of the tx.
b.page = nil
return b.Bucket(key), nil
}
// CreateBucketIfNotExists creates a new bucket if it doesn't already exist and returns a reference to it.
// Returns an error if the bucket name is blank, or if the bucket name is too long.
// The bucket instance is only valid for the lifetime of the transaction.
func (b *Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error) {
child, err := b.CreateBucket(key)
if err == ErrBucketExists {
return b.Bucket(key), nil
} else if err != nil {
return nil, err
}
return child, nil
}
// DeleteBucket deletes a bucket at the given key.
// Returns an error if the bucket does not exists, or if the key represents a non-bucket value.
func (b *Bucket) DeleteBucket(key []byte) error {
if b.tx.db == nil {
return ErrTxClosed
} else if !b.Writable() {
return ErrTxNotWritable
}
// Move cursor to correct position.
c := b.Cursor()
k, _, flags := c.seek(key)
// Return an error if bucket doesn't exist or is not a bucket.
if !bytes.Equal(key, k) {
return ErrBucketNotFound
} else if (flags & bucketLeafFlag) == 0 {
return ErrIncompatibleValue
}
// Recursively delete all child buckets.
child := b.Bucket(key)
err := child.ForEach(func(k, v []byte) error {
if v == nil {
if err := child.DeleteBucket(k); err != nil {
return fmt.Errorf("delete bucket: %s", err)
}
}
return nil
})
if err != nil {
return err
}
// Remove cached copy.
delete(b.buckets, string(key))
// Release all bucket pages to freelist.
child.nodes = nil
child.rootNode = nil
child.free()
// Delete the node if we have a matching key.
c.node().del(key)
return nil
}
// Get retrieves the value for a key in the bucket.
// Returns a nil value if the key does not exist or if the key is a nested bucket.
// The returned value is only valid for the life of the transaction.
func (b *Bucket) Get(key []byte) []byte {
k, v, flags := b.Cursor().seek(key)
// Return nil if this is a bucket.
if (flags & bucketLeafFlag) != 0 {
return nil
}
// If our target node isn't the same key as what's passed in then return nil.
if !bytes.Equal(key, k) {
return nil
}
return v
}
// Put sets the value for a key in the bucket.
// If the key exist then its previous value will be overwritten.
// Supplied value must remain valid for the life of the transaction.
// Returns an error if the bucket was created from a read-only transaction, if the key is blank, if the key is too large, or if the value is too large.
func (b *Bucket) Put(key []byte, value []byte) error {
if b.tx.db == nil {
return ErrTxClosed
} else if !b.Writable() {
return ErrTxNotWritable
} else if len(key) == 0 {
return ErrKeyRequired
} else if len(key) > MaxKeySize {
return ErrKeyTooLarge
} else if int64(len(value)) > MaxValueSize {
return ErrValueTooLarge
}
// Move cursor to correct position.
c := b.Cursor()
k, _, flags := c.seek(key)
// Return an error if there is an existing key with a bucket value.
if bytes.Equal(key, k) && (flags&bucketLeafFlag) != 0 {
return ErrIncompatibleValue
}
// Insert into node.
key = cloneBytes(key)
c.node().put(key, key, value, 0, 0)
return nil
}
// Delete removes a key from the bucket.
// If the key does not exist then nothing is done and a nil error is returned.
// Returns an error if the bucket was created from a read-only transaction.
func (b *Bucket) Delete(key []byte) error {
if b.tx.db == nil {
return ErrTxClosed
} else if !b.Writable() {
return ErrTxNotWritable
}
// Move cursor to correct position.
c := b.Cursor()
_, _, flags := c.seek(key)
// Return an error if there is already existing bucket value.
if (flags & bucketLeafFlag) != 0 {
return ErrIncompatibleValue
}
// Delete the node if we have a matching key.
c.node().del(key)
return nil
}
// Sequence returns the current integer for the bucket without incrementing it.
func (b *Bucket) Sequence() uint64 { return b.bucket.sequence }
// SetSequence updates the sequence number for the bucket.
func (b *Bucket) SetSequence(v uint64) error {
if b.tx.db == nil {
return ErrTxClosed
} else if !b.Writable() {
return ErrTxNotWritable
}
// Materialize the root node if it hasn't been already so that the
// bucket will be saved during commit.
if b.rootNode == nil {
_ = b.node(b.root, nil)
}
// Increment and return the sequence.
b.bucket.sequence = v
return nil
}
// NextSequence returns an autoincrementing integer for the bucket.
func (b *Bucket) NextSequence() (uint64, error) {
if b.tx.db == nil {
return 0, ErrTxClosed
} else if !b.Writable() {
return 0, ErrTxNotWritable
}
// Materialize the root node if it hasn't been already so that the
// bucket will be saved during commit.
if b.rootNode == nil {
_ = b.node(b.root, nil)
}
// Increment and return the sequence.
b.bucket.sequence++
return b.bucket.sequence, nil
}
// ForEach executes a function for each key/value pair in a bucket.
// If the provided function returns an error then the iteration is stopped and
// the error is returned to the caller. The provided function must not modify
// the bucket; this will result in undefined behavior.
func (b *Bucket) ForEach(fn func(k, v []byte) error) error {
if b.tx.db == nil {
return ErrTxClosed
}
c := b.Cursor()
for k, v := c.First(); k != nil; k, v = c.Next() {
if err := fn(k, v); err != nil {
return err
}
}
return nil
}
// Stat returns stats on a bucket.
func (b *Bucket) Stats() BucketStats {
var s, subStats BucketStats
pageSize := b.tx.db.pageSize
s.BucketN += 1
if b.root == 0 {
s.InlineBucketN += 1
}
b.forEachPage(func(p *page, depth int) {
if (p.flags & leafPageFlag) != 0 {
s.KeyN += int(p.count)
// used totals the used bytes for the page
used := pageHeaderSize
if p.count != 0 {
// If page has any elements, add all element headers.
used += leafPageElementSize * int(p.count-1)
// Add all element key, value sizes.
// The computation takes advantage of the fact that the position
// of the last element's key/value equals to the total of the sizes
// of all previous elements' keys and values.
// It also includes the last element's header.
lastElement := p.leafPageElement(p.count - 1)
used += int(lastElement.pos + lastElement.ksize + lastElement.vsize)
}
if b.root == 0 {
// For inlined bucket just update the inline stats
s.InlineBucketInuse += used
} else {
// For non-inlined bucket update all the leaf stats
s.LeafPageN++
s.LeafInuse += used
s.LeafOverflowN += int(p.overflow)
// Collect stats from sub-buckets.
// Do that by iterating over all element headers
// looking for the ones with the bucketLeafFlag.
for i := uint16(0); i < p.count; i++ {
e := p.leafPageElement(i)
if (e.flags & bucketLeafFlag) != 0 {
// For any bucket element, open the element value
// and recursively call Stats on the contained bucket.
subStats.Add(b.openBucket(e.value()).Stats())
}
}
}
} else if (p.flags & branchPageFlag) != 0 {
s.BranchPageN++
lastElement := p.branchPageElement(p.count - 1)
// used totals the used bytes for the page
// Add header and all element headers.
used := pageHeaderSize + (branchPageElementSize * int(p.count-1))
// Add size of all keys and values.
// Again, use the fact that last element's position equals to
// the total of key, value sizes of all previous elements.
used += int(lastElement.pos + lastElement.ksize)
s.BranchInuse += used
s.BranchOverflowN += int(p.overflow)
}
// Keep track of maximum page depth.
if depth+1 > s.Depth {
s.Depth = (depth + 1)
}
})
// Alloc stats can be computed from page counts and pageSize.
s.BranchAlloc = (s.BranchPageN + s.BranchOverflowN) * pageSize
s.LeafAlloc = (s.LeafPageN + s.LeafOverflowN) * pageSize
// Add the max depth of sub-buckets to get total nested depth.
s.Depth += subStats.Depth
// Add the stats for all sub-buckets
s.Add(subStats)
return s
}
// forEachPage iterates over every page in a bucket, including inline pages.
func (b *Bucket) forEachPage(fn func(*page, int)) {
// If we have an inline page then just use that.
if b.page != nil {
fn(b.page, 0)
return
}
// Otherwise traverse the page hierarchy.
b.tx.forEachPage(b.root, 0, fn)
}
// forEachPageNode iterates over every page (or node) in a bucket.
// This also includes inline pages.
func (b *Bucket) forEachPageNode(fn func(*page, *node, int)) {
// If we have an inline page or root node then just use that.
if b.page != nil {
fn(b.page, nil, 0)
return
}
b._forEachPageNode(b.root, 0, fn)
}
func (b *Bucket) _forEachPageNode(pgid pgid, depth int, fn func(*page, *node, int)) {
var p, n = b.pageNode(pgid)
// Execute function.
fn(p, n, depth)
// Recursively loop over children.
if p != nil {
if (p.flags & branchPageFlag) != 0 {
for i := 0; i < int(p.count); i++ {
elem := p.branchPageElement(uint16(i))
b._forEachPageNode(elem.pgid, depth+1, fn)
}
}
} else {
if !n.isLeaf {
for _, inode := range n.inodes {
b._forEachPageNode(inode.pgid, depth+1, fn)
}
}
}
}
// spill writes all the nodes for this bucket to dirty pages.
func (b *Bucket) spill() error {
// Spill all child buckets first.
for name, child := range b.buckets {
// If the child bucket is small enough and it has no child buckets then
// write it inline into the parent bucket's page. Otherwise spill it
// like a normal bucket and make the parent value a pointer to the page.
var value []byte
if child.inlineable() {
child.free()
value = child.write()
} else {
if err := child.spill(); err != nil {
return err
}
// Update the child bucket header in this bucket.
value = make([]byte, unsafe.Sizeof(bucket{}))
var bucket = (*bucket)(unsafe.Pointer(&value[0]))
*bucket = *child.bucket
}
// Skip writing the bucket if there are no materialized nodes.
if child.rootNode == nil {
continue
}
// Update parent node.
var c = b.Cursor()
k, _, flags := c.seek([]byte(name))
if !bytes.Equal([]byte(name), k) {
panic(fmt.Sprintf("misplaced bucket header: %x -> %x", []byte(name), k))
}
if flags&bucketLeafFlag == 0 {
panic(fmt.Sprintf("unexpected bucket header flag: %x", flags))
}
c.node().put([]byte(name), []byte(name), value, 0, bucketLeafFlag)
}
// Ignore if there's not a materialized root node.
if b.rootNode == nil {
return nil
}
// Spill nodes.
if err := b.rootNode.spill(); err != nil {
return err
}
b.rootNode = b.rootNode.root()
// Update the root node for this bucket.
if b.rootNode.pgid >= b.tx.meta.pgid {
panic(fmt.Sprintf("pgid (%d) above high water mark (%d)", b.rootNode.pgid, b.tx.meta.pgid))
}
b.root = b.rootNode.pgid
return nil
}
// inlineable returns true if a bucket is small enough to be written inline
// and if it contains no subbuckets. Otherwise returns false.
func (b *Bucket) inlineable() bool {
var n = b.rootNode
// Bucket must only contain a single leaf node.
if n == nil || !n.isLeaf {
return false
}
// Bucket is not inlineable if it contains subbuckets or if it goes beyond
// our threshold for inline bucket size.
var size = pageHeaderSize
for _, inode := range n.inodes {
size += leafPageElementSize + len(inode.key) + len(inode.value)
if inode.flags&bucketLeafFlag != 0 {
return false
} else if size > b.maxInlineBucketSize() {
return false
}
}
return true
}
// Returns the maximum total size of a bucket to make it a candidate for inlining.
func (b *Bucket) maxInlineBucketSize() int {
return b.tx.db.pageSize / 4
}
// write allocates and writes a bucket to a byte slice.
func (b *Bucket) write() []byte {
// Allocate the appropriate size.
var n = b.rootNode
var value = make([]byte, bucketHeaderSize+n.size())
// Write a bucket header.
var bucket = (*bucket)(unsafe.Pointer(&value[0]))
*bucket = *b.bucket
// Convert byte slice to a fake page and write the root node.
var p = (*page)(unsafe.Pointer(&value[bucketHeaderSize]))
n.write(p)
return value
}
// rebalance attempts to balance all nodes.
func (b *Bucket) rebalance() {
for _, n := range b.nodes {
n.rebalance()
}
for _, child := range b.buckets {
child.rebalance()
}
}
// node creates a node from a page and associates it with a given parent.
func (b *Bucket) node(pgid pgid, parent *node) *node {
_assert(b.nodes != nil, "nodes map expected")
// Retrieve node if it's already been created.
if n := b.nodes[pgid]; n != nil {
return n
}
// Otherwise create a node and cache it.
n := &node{bucket: b, parent: parent}
if parent == nil {
b.rootNode = n
} else {
parent.children = append(parent.children, n)
}
// Use the inline page if this is an inline bucket.
var p = b.page
if p == nil {
p = b.tx.page(pgid)
}
// Read the page into the node and cache it.
n.read(p)
b.nodes[pgid] = n
// Update statistics.
b.tx.stats.NodeCount++
return n
}
// free recursively frees all pages in the bucket.
func (b *Bucket) free() {
if b.root == 0 {
return
}
var tx = b.tx
b.forEachPageNode(func(p *page, n *node, _ int) {
if p != nil {
tx.db.freelist.free(tx.meta.txid, p)
} else {
n.free()
}
})
b.root = 0
}
// dereference removes all references to the old mmap.
func (b *Bucket) dereference() {
if b.rootNode != nil {
b.rootNode.root().dereference()
}
for _, child := range b.buckets {
child.dereference()
}
}
// pageNode returns the in-memory node, if it exists.
// Otherwise returns the underlying page.
func (b *Bucket) pageNode(id pgid) (*page, *node) {
// Inline buckets have a fake page embedded in their value so treat them
// differently. We'll return the rootNode (if available) or the fake page.
if b.root == 0 {
if id != 0 {
panic(fmt.Sprintf("inline bucket non-zero page access(2): %d != 0", id))
}
if b.rootNode != nil {
return nil, b.rootNode
}
return b.page, nil
}
// Check the node cache for non-inline buckets.
if b.nodes != nil {
if n := b.nodes[id]; n != nil {
return nil, n
}
}
// Finally lookup the page from the transaction if no node is materialized.
return b.tx.page(id), nil
}
// BucketStats records statistics about resources used by a bucket.
type BucketStats struct {
// Page count statistics.
BranchPageN int // number of logical branch pages
BranchOverflowN int // number of physical branch overflow pages
LeafPageN int // number of logical leaf pages
LeafOverflowN int // number of physical leaf overflow pages
// Tree statistics.
KeyN int // number of keys/value pairs
Depth int // number of levels in B+tree
// Page size utilization.
BranchAlloc int // bytes allocated for physical branch pages
BranchInuse int // bytes actually used for branch data
LeafAlloc int // bytes allocated for physical leaf pages
LeafInuse int // bytes actually used for leaf data
// Bucket statistics
BucketN int // total number of buckets including the top bucket
InlineBucketN int // total number on inlined buckets
InlineBucketInuse int // bytes used for inlined buckets (also accounted for in LeafInuse)
}
func (s *BucketStats) Add(other BucketStats) {
s.BranchPageN += other.BranchPageN
s.BranchOverflowN += other.BranchOverflowN
s.LeafPageN += other.LeafPageN
s.LeafOverflowN += other.LeafOverflowN
s.KeyN += other.KeyN
if s.Depth < other.Depth {
s.Depth = other.Depth
}
s.BranchAlloc += other.BranchAlloc
s.BranchInuse += other.BranchInuse
s.LeafAlloc += other.LeafAlloc
s.LeafInuse += other.LeafInuse
s.BucketN += other.BucketN
s.InlineBucketN += other.InlineBucketN
s.InlineBucketInuse += other.InlineBucketInuse
}
// cloneBytes returns a copy of a given slice.
func cloneBytes(v []byte) []byte {
var clone = make([]byte, len(v))
copy(clone, v)
return clone
}

400
vendor/github.com/boltdb/bolt/cursor.go generated vendored
View File

@ -1,400 +0,0 @@
package bolt
import (
"bytes"
"fmt"
"sort"
)
// Cursor represents an iterator that can traverse over all key/value pairs in a bucket in sorted order.
// Cursors see nested buckets with value == nil.
// Cursors can be obtained from a transaction and are valid as long as the transaction is open.
//
// Keys and values returned from the cursor are only valid for the life of the transaction.
//
// Changing data while traversing with a cursor may cause it to be invalidated
// and return unexpected keys and/or values. You must reposition your cursor
// after mutating data.
type Cursor struct {
bucket *Bucket
stack []elemRef
}
// Bucket returns the bucket that this cursor was created from.
func (c *Cursor) Bucket() *Bucket {
return c.bucket
}
// First moves the cursor to the first item in the bucket and returns its key and value.
// If the bucket is empty then a nil key and value are returned.
// The returned key and value are only valid for the life of the transaction.
func (c *Cursor) First() (key []byte, value []byte) {
_assert(c.bucket.tx.db != nil, "tx closed")
c.stack = c.stack[:0]
p, n := c.bucket.pageNode(c.bucket.root)
c.stack = append(c.stack, elemRef{page: p, node: n, index: 0})
c.first()
// If we land on an empty page then move to the next value.
// https://github.com/boltdb/bolt/issues/450
if c.stack[len(c.stack)-1].count() == 0 {
c.next()
}
k, v, flags := c.keyValue()
if (flags & uint32(bucketLeafFlag)) != 0 {
return k, nil
}
return k, v
}
// Last moves the cursor to the last item in the bucket and returns its key and value.
// If the bucket is empty then a nil key and value are returned.
// The returned key and value are only valid for the life of the transaction.
func (c *Cursor) Last() (key []byte, value []byte) {
_assert(c.bucket.tx.db != nil, "tx closed")
c.stack = c.stack[:0]
p, n := c.bucket.pageNode(c.bucket.root)
ref := elemRef{page: p, node: n}
ref.index = ref.count() - 1
c.stack = append(c.stack, ref)
c.last()
k, v, flags := c.keyValue()
if (flags & uint32(bucketLeafFlag)) != 0 {
return k, nil
}
return k, v
}
// Next moves the cursor to the next item in the bucket and returns its key and value.
// If the cursor is at the end of the bucket then a nil key and value are returned.
// The returned key and value are only valid for the life of the transaction.
func (c *Cursor) Next() (key []byte, value []byte) {
_assert(c.bucket.tx.db != nil, "tx closed")
k, v, flags := c.next()
if (flags & uint32(bucketLeafFlag)) != 0 {
return k, nil
}
return k, v
}
// Prev moves the cursor to the previous item in the bucket and returns its key and value.
// If the cursor is at the beginning of the bucket then a nil key and value are returned.
// The returned key and value are only valid for the life of the transaction.
func (c *Cursor) Prev() (key []byte, value []byte) {
_assert(c.bucket.tx.db != nil, "tx closed")
// Attempt to move back one element until we're successful.
// Move up the stack as we hit the beginning of each page in our stack.
for i := len(c.stack) - 1; i >= 0; i-- {
elem := &c.stack[i]
if elem.index > 0 {
elem.index--
break
}
c.stack = c.stack[:i]
}
// If we've hit the end then return nil.
if len(c.stack) == 0 {
return nil, nil
}
// Move down the stack to find the last element of the last leaf under this branch.
c.last()
k, v, flags := c.keyValue()
if (flags & uint32(bucketLeafFlag)) != 0 {
return k, nil
}
return k, v
}
// Seek moves the cursor to a given key and returns it.
// If the key does not exist then the next key is used. If no keys
// follow, a nil key is returned.
// The returned key and value are only valid for the life of the transaction.
func (c *Cursor) Seek(seek []byte) (key []byte, value []byte) {
k, v, flags := c.seek(seek)
// If we ended up after the last element of a page then move to the next one.
if ref := &c.stack[len(c.stack)-1]; ref.index >= ref.count() {
k, v, flags = c.next()
}
if k == nil {
return nil, nil
} else if (flags & uint32(bucketLeafFlag)) != 0 {
return k, nil
}
return k, v
}
// Delete removes the current key/value under the cursor from the bucket.
// Delete fails if current key/value is a bucket or if the transaction is not writable.
func (c *Cursor) Delete() error {
if c.bucket.tx.db == nil {
return ErrTxClosed
} else if !c.bucket.Writable() {
return ErrTxNotWritable
}
key, _, flags := c.keyValue()
// Return an error if current value is a bucket.
if (flags & bucketLeafFlag) != 0 {
return ErrIncompatibleValue
}
c.node().del(key)
return nil
}
// seek moves the cursor to a given key and returns it.
// If the key does not exist then the next key is used.
func (c *Cursor) seek(seek []byte) (key []byte, value []byte, flags uint32) {
_assert(c.bucket.tx.db != nil, "tx closed")
// Start from root page/node and traverse to correct page.
c.stack = c.stack[:0]
c.search(seek, c.bucket.root)
ref := &c.stack[len(c.stack)-1]
// If the cursor is pointing to the end of page/node then return nil.
if ref.index >= ref.count() {
return nil, nil, 0
}
// If this is a bucket then return a nil value.
return c.keyValue()
}
// first moves the cursor to the first leaf element under the last page in the stack.
func (c *Cursor) first() {
for {
// Exit when we hit a leaf page.
var ref = &c.stack[len(c.stack)-1]
if ref.isLeaf() {
break
}
// Keep adding pages pointing to the first element to the stack.
var pgid pgid
if ref.node != nil {
pgid = ref.node.inodes[ref.index].pgid
} else {
pgid = ref.page.branchPageElement(uint16(ref.index)).pgid
}
p, n := c.bucket.pageNode(pgid)
c.stack = append(c.stack, elemRef{page: p, node: n, index: 0})
}
}
// last moves the cursor to the last leaf element under the last page in the stack.
func (c *Cursor) last() {
for {
// Exit when we hit a leaf page.
ref := &c.stack[len(c.stack)-1]
if ref.isLeaf() {
break
}
// Keep adding pages pointing to the last element in the stack.
var pgid pgid
if ref.node != nil {
pgid = ref.node.inodes[ref.index].pgid
} else {
pgid = ref.page.branchPageElement(uint16(ref.index)).pgid
}
p, n := c.bucket.pageNode(pgid)
var nextRef = elemRef{page: p, node: n}
nextRef.index = nextRef.count() - 1
c.stack = append(c.stack, nextRef)
}
}
// next moves to the next leaf element and returns the key and value.
// If the cursor is at the last leaf element then it stays there and returns nil.
func (c *Cursor) next() (key []byte, value []byte, flags uint32) {
for {
// Attempt to move over one element until we're successful.
// Move up the stack as we hit the end of each page in our stack.
var i int
for i = len(c.stack) - 1; i >= 0; i-- {
elem := &c.stack[i]
if elem.index < elem.count()-1 {
elem.index++
break
}
}
// If we've hit the root page then stop and return. This will leave the
// cursor on the last element of the last page.
if i == -1 {
return nil, nil, 0
}
// Otherwise start from where we left off in the stack and find the
// first element of the first leaf page.
c.stack = c.stack[:i+1]
c.first()
// If this is an empty page then restart and move back up the stack.
// https://github.com/boltdb/bolt/issues/450
if c.stack[len(c.stack)-1].count() == 0 {
continue
}
return c.keyValue()
}
}
// search recursively performs a binary search against a given page/node until it finds a given key.
func (c *Cursor) search(key []byte, pgid pgid) {
p, n := c.bucket.pageNode(pgid)
if p != nil && (p.flags&(branchPageFlag|leafPageFlag)) == 0 {
panic(fmt.Sprintf("invalid page type: %d: %x", p.id, p.flags))
}
e := elemRef{page: p, node: n}
c.stack = append(c.stack, e)
// If we're on a leaf page/node then find the specific node.
if e.isLeaf() {
c.nsearch(key)
return
}
if n != nil {
c.searchNode(key, n)
return
}
c.searchPage(key, p)
}
func (c *Cursor) searchNode(key []byte, n *node) {
var exact bool
index := sort.Search(len(n.inodes), func(i int) bool {
// TODO(benbjohnson): Optimize this range search. It's a bit hacky right now.
// sort.Search() finds the lowest index where f() != -1 but we need the highest index.
ret := bytes.Compare(n.inodes[i].key, key)
if ret == 0 {
exact = true
}
return ret != -1
})
if !exact && index > 0 {
index--
}
c.stack[len(c.stack)-1].index = index
// Recursively search to the next page.
c.search(key, n.inodes[index].pgid)
}
func (c *Cursor) searchPage(key []byte, p *page) {
// Binary search for the correct range.
inodes := p.branchPageElements()
var exact bool
index := sort.Search(int(p.count), func(i int) bool {
// TODO(benbjohnson): Optimize this range search. It's a bit hacky right now.
// sort.Search() finds the lowest index where f() != -1 but we need the highest index.
ret := bytes.Compare(inodes[i].key(), key)
if ret == 0 {
exact = true
}
return ret != -1
})
if !exact && index > 0 {
index--
}
c.stack[len(c.stack)-1].index = index
// Recursively search to the next page.
c.search(key, inodes[index].pgid)
}
// nsearch searches the leaf node on the top of the stack for a key.
func (c *Cursor) nsearch(key []byte) {
e := &c.stack[len(c.stack)-1]
p, n := e.page, e.node
// If we have a node then search its inodes.
if n != nil {
index := sort.Search(len(n.inodes), func(i int) bool {
return bytes.Compare(n.inodes[i].key, key) != -1
})
e.index = index
return
}
// If we have a page then search its leaf elements.
inodes := p.leafPageElements()
index := sort.Search(int(p.count), func(i int) bool {
return bytes.Compare(inodes[i].key(), key) != -1
})
e.index = index
}
// keyValue returns the key and value of the current leaf element.
func (c *Cursor) keyValue() ([]byte, []byte, uint32) {
ref := &c.stack[len(c.stack)-1]
if ref.count() == 0 || ref.index >= ref.count() {
return nil, nil, 0
}
// Retrieve value from node.
if ref.node != nil {
inode := &ref.node.inodes[ref.index]
return inode.key, inode.value, inode.flags
}
// Or retrieve value from page.
elem := ref.page.leafPageElement(uint16(ref.index))
return elem.key(), elem.value(), elem.flags
}
// node returns the node that the cursor is currently positioned on.
func (c *Cursor) node() *node {
_assert(len(c.stack) > 0, "accessing a node with a zero-length cursor stack")
// If the top of the stack is a leaf node then just return it.
if ref := &c.stack[len(c.stack)-1]; ref.node != nil && ref.isLeaf() {
return ref.node
}
// Start from root and traverse down the hierarchy.
var n = c.stack[0].node
if n == nil {
n = c.bucket.node(c.stack[0].page.id, nil)
}
for _, ref := range c.stack[:len(c.stack)-1] {
_assert(!n.isLeaf, "expected branch node")
n = n.childAt(int(ref.index))
}
_assert(n.isLeaf, "expected leaf node")
return n
}
// elemRef represents a reference to an element on a given page/node.
type elemRef struct {
page *page
node *node
index int
}
// isLeaf returns whether the ref is pointing at a leaf page/node.
func (r *elemRef) isLeaf() bool {
if r.node != nil {
return r.node.isLeaf
}
return (r.page.flags & leafPageFlag) != 0
}
// count returns the number of inodes or page elements.
func (r *elemRef) count() int {
if r.node != nil {
return len(r.node.inodes)
}
return int(r.page.count)
}

1039
vendor/github.com/boltdb/bolt/db.go generated vendored
View File

File diff suppressed because it is too large Load Diff

44
vendor/github.com/boltdb/bolt/doc.go generated vendored
View File

@ -1,44 +0,0 @@
/*
Package bolt implements a low-level key/value store in pure Go. It supports
fully serializable transactions, ACID semantics, and lock-free MVCC with
multiple readers and a single writer. Bolt can be used for projects that
want a simple data store without the need to add large dependencies such as
Postgres or MySQL.
Bolt is a single-level, zero-copy, B+tree data store. This means that Bolt is
optimized for fast read access and does not require recovery in the event of a
system crash. Transactions which have not finished committing will simply be
rolled back in the event of a crash.
The design of Bolt is based on Howard Chu's LMDB database project.
Bolt currently works on Windows, Mac OS X, and Linux.
Basics
There are only a few types in Bolt: DB, Bucket, Tx, and Cursor. The DB is
a collection of buckets and is represented by a single file on disk. A bucket is
a collection of unique keys that are associated with values.
Transactions provide either read-only or read-write access to the database.
Read-only transactions can retrieve key/value pairs and can use Cursors to
iterate over the dataset sequentially. Read-write transactions can create and
delete buckets and can insert and remove keys. Only one read-write transaction
is allowed at a time.
Caveats
The database uses a read-only, memory-mapped data file to ensure that
applications cannot corrupt the database, however, this means that keys and
values returned from Bolt cannot be changed. Writing to a read-only byte slice
will cause Go to panic.
Keys and values retrieved from the database are only valid for the life of
the transaction. When used outside the transaction, these byte slices can
point to different data or can point to invalid memory which will cause a panic.
*/
package bolt

71
vendor/github.com/boltdb/bolt/errors.go generated vendored
View File

@ -1,71 +0,0 @@
package bolt
import "errors"
// These errors can be returned when opening or calling methods on a DB.
var (
// ErrDatabaseNotOpen is returned when a DB instance is accessed before it
// is opened or after it is closed.
ErrDatabaseNotOpen = errors.New("database not open")
// ErrDatabaseOpen is returned when opening a database that is
// already open.
ErrDatabaseOpen = errors.New("database already open")
// ErrInvalid is returned when both meta pages on a database are invalid.
// This typically occurs when a file is not a bolt database.
ErrInvalid = errors.New("invalid database")
// ErrVersionMismatch is returned when the data file was created with a
// different version of Bolt.
ErrVersionMismatch = errors.New("version mismatch")
// ErrChecksum is returned when either meta page checksum does not match.
ErrChecksum = errors.New("checksum error")
// ErrTimeout is returned when a database cannot obtain an exclusive lock
// on the data file after the timeout passed to Open().
ErrTimeout = errors.New("timeout")
)
// These errors can occur when beginning or committing a Tx.
var (
// ErrTxNotWritable is returned when performing a write operation on a
// read-only transaction.
ErrTxNotWritable = errors.New("tx not writable")
// ErrTxClosed is returned when committing or rolling back a transaction
// that has already been committed or rolled back.
ErrTxClosed = errors.New("tx closed")
// ErrDatabaseReadOnly is returned when a mutating transaction is started on a
// read-only database.
ErrDatabaseReadOnly = errors.New("database is in read-only mode")
)
// These errors can occur when putting or deleting a value or a bucket.
var (
// ErrBucketNotFound is returned when trying to access a bucket that has
// not been created yet.
ErrBucketNotFound = errors.New("bucket not found")
// ErrBucketExists is returned when creating a bucket that already exists.
ErrBucketExists = errors.New("bucket already exists")
// ErrBucketNameRequired is returned when creating a bucket with a blank name.
ErrBucketNameRequired = errors.New("bucket name required")
// ErrKeyRequired is returned when inserting a zero-length key.
ErrKeyRequired = errors.New("key required")
// ErrKeyTooLarge is returned when inserting a key that is larger than MaxKeySize.
ErrKeyTooLarge = errors.New("key too large")
// ErrValueTooLarge is returned when inserting a value that is larger than MaxValueSize.
ErrValueTooLarge = errors.New("value too large")
// ErrIncompatibleValue is returned when trying create or delete a bucket
// on an existing non-bucket key or when trying to create or delete a
// non-bucket key on an existing bucket key.
ErrIncompatibleValue = errors.New("incompatible value")
)

252
vendor/github.com/boltdb/bolt/freelist.go generated vendored
View File

@ -1,252 +0,0 @@
package bolt
import (
"fmt"
"sort"
"unsafe"
)
// freelist represents a list of all pages that are available for allocation.
// It also tracks pages that have been freed but are still in use by open transactions.
type freelist struct {
ids []pgid // all free and available free page ids.
pending map[txid][]pgid // mapping of soon-to-be free page ids by tx.
cache map[pgid]bool // fast lookup of all free and pending page ids.
}
// newFreelist returns an empty, initialized freelist.
func newFreelist() *freelist {
return &freelist{
pending: make(map[txid][]pgid),
cache: make(map[pgid]bool),
}
}
// size returns the size of the page after serialization.
func (f *freelist) size() int {
n := f.count()
if n >= 0xFFFF {
// The first element will be used to store the count. See freelist.write.
n++
}
return pageHeaderSize + (int(unsafe.Sizeof(pgid(0))) * n)
}
// count returns count of pages on the freelist
func (f *freelist) count() int {
return f.free_count() + f.pending_count()
}
// free_count returns count of free pages
func (f *freelist) free_count() int {
return len(f.ids)
}
// pending_count returns count of pending pages
func (f *freelist) pending_count() int {
var count int
for _, list := range f.pending {
count += len(list)
}
return count
}
// copyall copies into dst a list of all free ids and all pending ids in one sorted list.
// f.count returns the minimum length required for dst.
func (f *freelist) copyall(dst []pgid) {
m := make(pgids, 0, f.pending_count())
for _, list := range f.pending {
m = append(m, list...)
}
sort.Sort(m)
mergepgids(dst, f.ids, m)
}
// allocate returns the starting page id of a contiguous list of pages of a given size.
// If a contiguous block cannot be found then 0 is returned.
func (f *freelist) allocate(n int) pgid {
if len(f.ids) == 0 {
return 0
}
var initial, previd pgid
for i, id := range f.ids {
if id <= 1 {
panic(fmt.Sprintf("invalid page allocation: %d", id))
}
// Reset initial page if this is not contiguous.
if previd == 0 || id-previd != 1 {
initial = id
}
// If we found a contiguous block then remove it and return it.
if (id-initial)+1 == pgid(n) {
// If we're allocating off the beginning then take the fast path
// and just adjust the existing slice. This will use extra memory
// temporarily but the append() in free() will realloc the slice
// as is necessary.
if (i + 1) == n {
f.ids = f.ids[i+1:]
} else {
copy(f.ids[i-n+1:], f.ids[i+1:])
f.ids = f.ids[:len(f.ids)-n]
}
// Remove from the free cache.
for i := pgid(0); i < pgid(n); i++ {
delete(f.cache, initial+i)
}
return initial
}
previd = id
}
return 0
}
// free releases a page and its overflow for a given transaction id.
// If the page is already free then a panic will occur.
func (f *freelist) free(txid txid, p *page) {
if p.id <= 1 {
panic(fmt.Sprintf("cannot free page 0 or 1: %d", p.id))
}
// Free page and all its overflow pages.
var ids = f.pending[txid]
for id := p.id; id <= p.id+pgid(p.overflow); id++ {
// Verify that page is not already free.
if f.cache[id] {
panic(fmt.Sprintf("page %d already freed", id))
}
// Add to the freelist and cache.
ids = append(ids, id)
f.cache[id] = true
}
f.pending[txid] = ids
}
// release moves all page ids for a transaction id (or older) to the freelist.
func (f *freelist) release(txid txid) {
m := make(pgids, 0)
for tid, ids := range f.pending {
if tid <= txid {
// Move transaction's pending pages to the available freelist.
// Don't remove from the cache since the page is still free.
m = append(m, ids...)
delete(f.pending, tid)
}
}
sort.Sort(m)
f.ids = pgids(f.ids).merge(m)
}
// rollback removes the pages from a given pending tx.
func (f *freelist) rollback(txid txid) {
// Remove page ids from cache.
for _, id := range f.pending[txid] {
delete(f.cache, id)
}
// Remove pages from pending list.
delete(f.pending, txid)
}
// freed returns whether a given page is in the free list.
func (f *freelist) freed(pgid pgid) bool {
return f.cache[pgid]
}
// read initializes the freelist from a freelist page.
func (f *freelist) read(p *page) {
// If the page.count is at the max uint16 value (64k) then it's considered
// an overflow and the size of the freelist is stored as the first element.
idx, count := 0, int(p.count)
if count == 0xFFFF {
idx = 1
count = int(((*[maxAllocSize]pgid)(unsafe.Pointer(&p.ptr)))[0])
}
// Copy the list of page ids from the freelist.
if count == 0 {
f.ids = nil
} else {
ids := ((*[maxAllocSize]pgid)(unsafe.Pointer(&p.ptr)))[idx:count]
f.ids = make([]pgid, len(ids))
copy(f.ids, ids)
// Make sure they're sorted.
sort.Sort(pgids(f.ids))
}
// Rebuild the page cache.
f.reindex()
}
// write writes the page ids onto a freelist page. All free and pending ids are
// saved to disk since in the event of a program crash, all pending ids will
// become free.
func (f *freelist) write(p *page) error {
// Combine the old free pgids and pgids waiting on an open transaction.
// Update the header flag.
p.flags |= freelistPageFlag
// The page.count can only hold up to 64k elements so if we overflow that
// number then we handle it by putting the size in the first element.
lenids := f.count()
if lenids == 0 {
p.count = uint16(lenids)
} else if lenids < 0xFFFF {
p.count = uint16(lenids)
f.copyall(((*[maxAllocSize]pgid)(unsafe.Pointer(&p.ptr)))[:])
} else {
p.count = 0xFFFF
((*[maxAllocSize]pgid)(unsafe.Pointer(&p.ptr)))[0] = pgid(lenids)
f.copyall(((*[maxAllocSize]pgid)(unsafe.Pointer(&p.ptr)))[1:])
}
return nil
}
// reload reads the freelist from a page and filters out pending items.
func (f *freelist) reload(p *page) {
f.read(p)
// Build a cache of only pending pages.
pcache := make(map[pgid]bool)
for _, pendingIDs := range f.pending {
for _, pendingID := range pendingIDs {
pcache[pendingID] = true
}
}
// Check each page in the freelist and build a new available freelist
// with any pages not in the pending lists.
var a []pgid
for _, id := range f.ids {
if !pcache[id] {
a = append(a, id)
}
}
f.ids = a
// Once the available list is rebuilt then rebuild the free cache so that
// it includes the available and pending free pages.
f.reindex()
}
// reindex rebuilds the free cache based on available and pending free lists.
func (f *freelist) reindex() {
f.cache = make(map[pgid]bool, len(f.ids))
for _, id := range f.ids {
f.cache[id] = true
}
for _, pendingIDs := range f.pending {
for _, pendingID := range pendingIDs {
f.cache[pendingID] = true
}
}
}

604
vendor/github.com/boltdb/bolt/node.go generated vendored
View File

@ -1,604 +0,0 @@
package bolt
import (
"bytes"
"fmt"
"sort"
"unsafe"
)
// node represents an in-memory, deserialized page.
type node struct {
bucket *Bucket
isLeaf bool
unbalanced bool
spilled bool
key []byte
pgid pgid
parent *node
children nodes
inodes inodes
}
// root returns the top-level node this node is attached to.
func (n *node) root() *node {
if n.parent == nil {
return n
}
return n.parent.root()
}
// minKeys returns the minimum number of inodes this node should have.
func (n *node) minKeys() int {
if n.isLeaf {
return 1
}
return 2
}
// size returns the size of the node after serialization.
func (n *node) size() int {
sz, elsz := pageHeaderSize, n.pageElementSize()
for i := 0; i < len(n.inodes); i++ {
item := &n.inodes[i]
sz += elsz + len(item.key) + len(item.value)
}
return sz
}
// sizeLessThan returns true if the node is less than a given size.
// This is an optimization to avoid calculating a large node when we only need
// to know if it fits inside a certain page size.
func (n *node) sizeLessThan(v int) bool {
sz, elsz := pageHeaderSize, n.pageElementSize()
for i := 0; i < len(n.inodes); i++ {
item := &n.inodes[i]
sz += elsz + len(item.key) + len(item.value)
if sz >= v {
return false
}
}
return true
}
// pageElementSize returns the size of each page element based on the type of node.
func (n *node) pageElementSize() int {
if n.isLeaf {
return leafPageElementSize
}
return branchPageElementSize
}
// childAt returns the child node at a given index.
func (n *node) childAt(index int) *node {
if n.isLeaf {
panic(fmt.Sprintf("invalid childAt(%d) on a leaf node", index))
}
return n.bucket.node(n.inodes[index].pgid, n)
}
// childIndex returns the index of a given child node.
func (n *node) childIndex(child *node) int {
index := sort.Search(len(n.inodes), func(i int) bool { return bytes.Compare(n.inodes[i].key, child.key) != -1 })
return index
}
// numChildren returns the number of children.
func (n *node) numChildren() int {
return len(n.inodes)
}
// nextSibling returns the next node with the same parent.
func (n *node) nextSibling() *node {
if n.parent == nil {
return nil
}
index := n.parent.childIndex(n)
if index >= n.parent.numChildren()-1 {
return nil
}
return n.parent.childAt(index + 1)
}
// prevSibling returns the previous node with the same parent.
func (n *node) prevSibling() *node {
if n.parent == nil {
return nil
}
index := n.parent.childIndex(n)
if index == 0 {
return nil
}
return n.parent.childAt(index - 1)
}
// put inserts a key/value.
func (n *node) put(oldKey, newKey, value []byte, pgid pgid, flags uint32) {
if pgid >= n.bucket.tx.meta.pgid {
panic(fmt.Sprintf("pgid (%d) above high water mark (%d)", pgid, n.bucket.tx.meta.pgid))
} else if len(oldKey) <= 0 {
panic("put: zero-length old key")
} else if len(newKey) <= 0 {
panic("put: zero-length new key")
}
// Find insertion index.
index := sort.Search(len(n.inodes), func(i int) bool { return bytes.Compare(n.inodes[i].key, oldKey) != -1 })
// Add capacity and shift nodes if we don't have an exact match and need to insert.
exact := (len(n.inodes) > 0 && index < len(n.inodes) && bytes.Equal(n.inodes[index].key, oldKey))
if !exact {
n.inodes = append(n.inodes, inode{})
copy(n.inodes[index+1:], n.inodes[index:])
}
inode := &n.inodes[index]
inode.flags = flags
inode.key = newKey
inode.value = value
inode.pgid = pgid
_assert(len(inode.key) > 0, "put: zero-length inode key")
}
// del removes a key from the node.
func (n *node) del(key []byte) {
// Find index of key.
index := sort.Search(len(n.inodes), func(i int) bool { return bytes.Compare(n.inodes[i].key, key) != -1 })
// Exit if the key isn't found.
if index >= len(n.inodes) || !bytes.Equal(n.inodes[index].key, key) {
return
}
// Delete inode from the node.
n.inodes = append(n.inodes[:index], n.inodes[index+1:]...)
// Mark the node as needing rebalancing.
n.unbalanced = true
}
// read initializes the node from a page.
func (n *node) read(p *page) {
n.pgid = p.id
n.isLeaf = ((p.flags & leafPageFlag) != 0)
n.inodes = make(inodes, int(p.count))
for i := 0; i < int(p.count); i++ {
inode := &n.inodes[i]
if n.isLeaf {
elem := p.leafPageElement(uint16(i))
inode.flags = elem.flags
inode.key = elem.key()
inode.value = elem.value()
} else {
elem := p.branchPageElement(uint16(i))
inode.pgid = elem.pgid
inode.key = elem.key()
}
_assert(len(inode.key) > 0, "read: zero-length inode key")
}
// Save first key so we can find the node in the parent when we spill.
if len(n.inodes) > 0 {
n.key = n.inodes[0].key
_assert(len(n.key) > 0, "read: zero-length node key")
} else {
n.key = nil
}
}
// write writes the items onto one or more pages.
func (n *node) write(p *page) {
// Initialize page.
if n.isLeaf {
p.flags |= leafPageFlag
} else {
p.flags |= branchPageFlag
}
if len(n.inodes) >= 0xFFFF {
panic(fmt.Sprintf("inode overflow: %d (pgid=%d)", len(n.inodes), p.id))
}
p.count = uint16(len(n.inodes))
// Stop here if there are no items to write.
if p.count == 0 {
return
}
// Loop over each item and write it to the page.
b := (*[maxAllocSize]byte)(unsafe.Pointer(&p.ptr))[n.pageElementSize()*len(n.inodes):]
for i, item := range n.inodes {
_assert(len(item.key) > 0, "write: zero-length inode key")
// Write the page element.
if n.isLeaf {
elem := p.leafPageElement(uint16(i))
elem.pos = uint32(uintptr(unsafe.Pointer(&b[0])) - uintptr(unsafe.Pointer(elem)))
elem.flags = item.flags
elem.ksize = uint32(len(item.key))
elem.vsize = uint32(len(item.value))
} else {
elem := p.branchPageElement(uint16(i))
elem.pos = uint32(uintptr(unsafe.Pointer(&b[0])) - uintptr(unsafe.Pointer(elem)))
elem.ksize = uint32(len(item.key))
elem.pgid = item.pgid
_assert(elem.pgid != p.id, "write: circular dependency occurred")
}
// If the length of key+value is larger than the max allocation size
// then we need to reallocate the byte array pointer.
//
// See: https://github.com/boltdb/bolt/pull/335
klen, vlen := len(item.key), len(item.value)
if len(b) < klen+vlen {
b = (*[maxAllocSize]byte)(unsafe.Pointer(&b[0]))[:]
}
// Write data for the element to the end of the page.
copy(b[0:], item.key)
b = b[klen:]
copy(b[0:], item.value)
b = b[vlen:]
}
// DEBUG ONLY: n.dump()
}
// split breaks up a node into multiple smaller nodes, if appropriate.
// This should only be called from the spill() function.
func (n *node) split(pageSize int) []*node {
var nodes []*node
node := n
for {
// Split node into two.
a, b := node.splitTwo(pageSize)
nodes = append(nodes, a)
// If we can't split then exit the loop.
if b == nil {
break
}
// Set node to b so it gets split on the next iteration.
node = b
}
return nodes
}
// splitTwo breaks up a node into two smaller nodes, if appropriate.
// This should only be called from the split() function.
func (n *node) splitTwo(pageSize int) (*node, *node) {
// Ignore the split if the page doesn't have at least enough nodes for
// two pages or if the nodes can fit in a single page.
if len(n.inodes) <= (minKeysPerPage*2) || n.sizeLessThan(pageSize) {
return n, nil
}
// Determine the threshold before starting a new node.
var fillPercent = n.bucket.FillPercent
if fillPercent < minFillPercent {
fillPercent = minFillPercent
} else if fillPercent > maxFillPercent {
fillPercent = maxFillPercent
}
threshold := int(float64(pageSize) * fillPercent)
// Determine split position and sizes of the two pages.
splitIndex, _ := n.splitIndex(threshold)
// Split node into two separate nodes.
// If there's no parent then we'll need to create one.
if n.parent == nil {
n.parent = &node{bucket: n.bucket, children: []*node{n}}
}
// Create a new node and add it to the parent.
next := &node{bucket: n.bucket, isLeaf: n.isLeaf, parent: n.parent}
n.parent.children = append(n.parent.children, next)
// Split inodes across two nodes.
next.inodes = n.inodes[splitIndex:]
n.inodes = n.inodes[:splitIndex]
// Update the statistics.
n.bucket.tx.stats.Split++
return n, next
}
// splitIndex finds the position where a page will fill a given threshold.
// It returns the index as well as the size of the first page.
// This is only be called from split().
func (n *node) splitIndex(threshold int) (index, sz int) {
sz = pageHeaderSize
// Loop until we only have the minimum number of keys required for the second page.
for i := 0; i < len(n.inodes)-minKeysPerPage; i++ {
index = i
inode := n.inodes[i]
elsize := n.pageElementSize() + len(inode.key) + len(inode.value)
// If we have at least the minimum number of keys and adding another
// node would put us over the threshold then exit and return.
if i >= minKeysPerPage && sz+elsize > threshold {
break
}
// Add the element size to the total size.
sz += elsize
}
return
}
// spill writes the nodes to dirty pages and splits nodes as it goes.
// Returns an error if dirty pages cannot be allocated.
func (n *node) spill() error {
var tx = n.bucket.tx
if n.spilled {
return nil
}
// Spill child nodes first. Child nodes can materialize sibling nodes in
// the case of split-merge so we cannot use a range loop. We have to check
// the children size on every loop iteration.
sort.Sort(n.children)
for i := 0; i < len(n.children); i++ {
if err := n.children[i].spill(); err != nil {
return err
}
}
// We no longer need the child list because it's only used for spill tracking.
n.children = nil
// Split nodes into appropriate sizes. The first node will always be n.
var nodes = n.split(tx.db.pageSize)
for _, node := range nodes {
// Add node's page to the freelist if it's not new.
if node.pgid > 0 {
tx.db.freelist.free(tx.meta.txid, tx.page(node.pgid))
node.pgid = 0
}
// Allocate contiguous space for the node.
p, err := tx.allocate((node.size() / tx.db.pageSize) + 1)
if err != nil {
return err
}
// Write the node.
if p.id >= tx.meta.pgid {
panic(fmt.Sprintf("pgid (%d) above high water mark (%d)", p.id, tx.meta.pgid))
}
node.pgid = p.id
node.write(p)
node.spilled = true
// Insert into parent inodes.
if node.parent != nil {
var key = node.key
if key == nil {
key = node.inodes[0].key
}
node.parent.put(key, node.inodes[0].key, nil, node.pgid, 0)
node.key = node.inodes[0].key
_assert(len(node.key) > 0, "spill: zero-length node key")
}
// Update the statistics.
tx.stats.Spill++
}
// If the root node split and created a new root then we need to spill that
// as well. We'll clear out the children to make sure it doesn't try to respill.
if n.parent != nil && n.parent.pgid == 0 {
n.children = nil
return n.parent.spill()
}
return nil
}
// rebalance attempts to combine the node with sibling nodes if the node fill
// size is below a threshold or if there are not enough keys.
func (n *node) rebalance() {
if !n.unbalanced {
return
}
n.unbalanced = false
// Update statistics.
n.bucket.tx.stats.Rebalance++
// Ignore if node is above threshold (25%) and has enough keys.
var threshold = n.bucket.tx.db.pageSize / 4
if n.size() > threshold && len(n.inodes) > n.minKeys() {
return
}
// Root node has special handling.
if n.parent == nil {
// If root node is a branch and only has one node then collapse it.
if !n.isLeaf && len(n.inodes) == 1 {
// Move root's child up.
child := n.bucket.node(n.inodes[0].pgid, n)
n.isLeaf = child.isLeaf
n.inodes = child.inodes[:]
n.children = child.children
// Reparent all child nodes being moved.
for _, inode := range n.inodes {
if child, ok := n.bucket.nodes[inode.pgid]; ok {
child.parent = n
}
}
// Remove old child.
child.parent = nil
delete(n.bucket.nodes, child.pgid)
child.free()
}
return
}
// If node has no keys then just remove it.
if n.numChildren() == 0 {
n.parent.del(n.key)
n.parent.removeChild(n)
delete(n.bucket.nodes, n.pgid)
n.free()
n.parent.rebalance()
return
}
_assert(n.parent.numChildren() > 1, "parent must have at least 2 children")
// Destination node is right sibling if idx == 0, otherwise left sibling.
var target *node
var useNextSibling = (n.parent.childIndex(n) == 0)
if useNextSibling {
target = n.nextSibling()
} else {
target = n.prevSibling()
}
// If both this node and the target node are too small then merge them.
if useNextSibling {
// Reparent all child nodes being moved.
for _, inode := range target.inodes {
if child, ok := n.bucket.nodes[inode.pgid]; ok {
child.parent.removeChild(child)
child.parent = n
child.parent.children = append(child.parent.children, child)
}
}
// Copy over inodes from target and remove target.
n.inodes = append(n.inodes, target.inodes...)
n.parent.del(target.key)
n.parent.removeChild(target)
delete(n.bucket.nodes, target.pgid)
target.free()
} else {
// Reparent all child nodes being moved.
for _, inode := range n.inodes {
if child, ok := n.bucket.nodes[inode.pgid]; ok {
child.parent.removeChild(child)
child.parent = target
child.parent.children = append(child.parent.children, child)
}
}
// Copy over inodes to target and remove node.
target.inodes = append(target.inodes, n.inodes...)
n.parent.del(n.key)
n.parent.removeChild(n)
delete(n.bucket.nodes, n.pgid)
n.free()
}
// Either this node or the target node was deleted from the parent so rebalance it.
n.parent.rebalance()
}
// removes a node from the list of in-memory children.
// This does not affect the inodes.
func (n *node) removeChild(target *node) {
for i, child := range n.children {
if child == target {
n.children = append(n.children[:i], n.children[i+1:]...)
return
}
}
}
// dereference causes the node to copy all its inode key/value references to heap memory.
// This is required when the mmap is reallocated so inodes are not pointing to stale data.
func (n *node) dereference() {
if n.key != nil {
key := make([]byte, len(n.key))
copy(key, n.key)
n.key = key
_assert(n.pgid == 0 || len(n.key) > 0, "dereference: zero-length node key on existing node")
}
for i := range n.inodes {
inode := &n.inodes[i]
key := make([]byte, len(inode.key))
copy(key, inode.key)
inode.key = key
_assert(len(inode.key) > 0, "dereference: zero-length inode key")
value := make([]byte, len(inode.value))
copy(value, inode.value)
inode.value = value
}
// Recursively dereference children.
for _, child := range n.children {
child.dereference()
}
// Update statistics.
n.bucket.tx.stats.NodeDeref++
}
// free adds the node's underlying page to the freelist.
func (n *node) free() {
if n.pgid != 0 {
n.bucket.tx.db.freelist.free(n.bucket.tx.meta.txid, n.bucket.tx.page(n.pgid))
n.pgid = 0
}
}
// dump writes the contents of the node to STDERR for debugging purposes.
/*
func (n *node) dump() {
// Write node header.
var typ = "branch"
if n.isLeaf {
typ = "leaf"
}
warnf("[NODE %d {type=%s count=%d}]", n.pgid, typ, len(n.inodes))
// Write out abbreviated version of each item.
for _, item := range n.inodes {
if n.isLeaf {
if item.flags&bucketLeafFlag != 0 {
bucket := (*bucket)(unsafe.Pointer(&item.value[0]))
warnf("+L %08x -> (bucket root=%d)", trunc(item.key, 4), bucket.root)
} else {
warnf("+L %08x -> %08x", trunc(item.key, 4), trunc(item.value, 4))
}
} else {
warnf("+B %08x -> pgid=%d", trunc(item.key, 4), item.pgid)
}
}
warn("")
}
*/
type nodes []*node
func (s nodes) Len() int { return len(s) }
func (s nodes) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
func (s nodes) Less(i, j int) bool { return bytes.Compare(s[i].inodes[0].key, s[j].inodes[0].key) == -1 }
// inode represents an internal node inside of a node.
// It can be used to point to elements in a page or point
// to an element which hasn't been added to a page yet.
type inode struct {
flags uint32
pgid pgid
key []byte
value []byte
}
type inodes []inode

197
vendor/github.com/boltdb/bolt/page.go generated vendored
View File

@ -1,197 +0,0 @@
package bolt
import (
"fmt"
"os"
"sort"
"unsafe"
)
const pageHeaderSize = int(unsafe.Offsetof(((*page)(nil)).ptr))
const minKeysPerPage = 2
const branchPageElementSize = int(unsafe.Sizeof(branchPageElement{}))
const leafPageElementSize = int(unsafe.Sizeof(leafPageElement{}))
const (
branchPageFlag = 0x01
leafPageFlag = 0x02
metaPageFlag = 0x04
freelistPageFlag = 0x10
)
const (
bucketLeafFlag = 0x01
)
type pgid uint64
type page struct {
id pgid
flags uint16
count uint16
overflow uint32
ptr uintptr
}
// typ returns a human readable page type string used for debugging.
func (p *page) typ() string {
if (p.flags & branchPageFlag) != 0 {
return "branch"
} else if (p.flags & leafPageFlag) != 0 {
return "leaf"
} else if (p.flags & metaPageFlag) != 0 {
return "meta"
} else if (p.flags & freelistPageFlag) != 0 {
return "freelist"
}
return fmt.Sprintf("unknown<%02x>", p.flags)
}
// meta returns a pointer to the metadata section of the page.
func (p *page) meta() *meta {
return (*meta)(unsafe.Pointer(&p.ptr))
}
// leafPageElement retrieves the leaf node by index
func (p *page) leafPageElement(index uint16) *leafPageElement {
n := &((*[0x7FFFFFF]leafPageElement)(unsafe.Pointer(&p.ptr)))[index]
return n
}
// leafPageElements retrieves a list of leaf nodes.
func (p *page) leafPageElements() []leafPageElement {
if p.count == 0 {
return nil
}
return ((*[0x7FFFFFF]leafPageElement)(unsafe.Pointer(&p.ptr)))[:]
}
// branchPageElement retrieves the branch node by index
func (p *page) branchPageElement(index uint16) *branchPageElement {
return &((*[0x7FFFFFF]branchPageElement)(unsafe.Pointer(&p.ptr)))[index]
}
// branchPageElements retrieves a list of branch nodes.
func (p *page) branchPageElements() []branchPageElement {
if p.count == 0 {
return nil
}
return ((*[0x7FFFFFF]branchPageElement)(unsafe.Pointer(&p.ptr)))[:]
}
// dump writes n bytes of the page to STDERR as hex output.
func (p *page) hexdump(n int) {
buf := (*[maxAllocSize]byte)(unsafe.Pointer(p))[:n]
fmt.Fprintf(os.Stderr, "%x\n", buf)
}
type pages []*page
func (s pages) Len() int { return len(s) }
func (s pages) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
func (s pages) Less(i, j int) bool { return s[i].id < s[j].id }
// branchPageElement represents a node on a branch page.
type branchPageElement struct {
pos uint32
ksize uint32
pgid pgid
}
// key returns a byte slice of the node key.
func (n *branchPageElement) key() []byte {
buf := (*[maxAllocSize]byte)(unsafe.Pointer(n))
return (*[maxAllocSize]byte)(unsafe.Pointer(&buf[n.pos]))[:n.ksize]
}
// leafPageElement represents a node on a leaf page.
type leafPageElement struct {
flags uint32
pos uint32
ksize uint32
vsize uint32
}
// key returns a byte slice of the node key.
func (n *leafPageElement) key() []byte {
buf := (*[maxAllocSize]byte)(unsafe.Pointer(n))
return (*[maxAllocSize]byte)(unsafe.Pointer(&buf[n.pos]))[:n.ksize:n.ksize]
}
// value returns a byte slice of the node value.
func (n *leafPageElement) value() []byte {
buf := (*[maxAllocSize]byte)(unsafe.Pointer(n))
return (*[maxAllocSize]byte)(unsafe.Pointer(&buf[n.pos+n.ksize]))[:n.vsize:n.vsize]
}
// PageInfo represents human readable information about a page.
type PageInfo struct {
ID int
Type string
Count int
OverflowCount int
}
type pgids []pgid
func (s pgids) Len() int { return len(s) }
func (s pgids) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
func (s pgids) Less(i, j int) bool { return s[i] < s[j] }
// merge returns the sorted union of a and b.
func (a pgids) merge(b pgids) pgids {
// Return the opposite slice if one is nil.
if len(a) == 0 {
return b
}
if len(b) == 0 {
return a
}
merged := make(pgids, len(a)+len(b))
mergepgids(merged, a, b)
return merged
}
// mergepgids copies the sorted union of a and b into dst.
// If dst is too small, it panics.
func mergepgids(dst, a, b pgids) {
if len(dst) < len(a)+len(b) {
panic(fmt.Errorf("mergepgids bad len %d < %d + %d", len(dst), len(a), len(b)))
}
// Copy in the opposite slice if one is nil.
if len(a) == 0 {
copy(dst, b)
return
}
if len(b) == 0 {
copy(dst, a)
return
}
// Merged will hold all elements from both lists.
merged := dst[:0]
// Assign lead to the slice with a lower starting value, follow to the higher value.
lead, follow := a, b
if b[0] < a[0] {
lead, follow = b, a
}
// Continue while there are elements in the lead.
for len(lead) > 0 {
// Merge largest prefix of lead that is ahead of follow[0].
n := sort.Search(len(lead), func(i int) bool { return lead[i] > follow[0] })
merged = append(merged, lead[:n]...)
if n >= len(lead) {
break
}
// Swap lead and follow.
lead, follow = follow, lead[n:]
}
// Append what's left in follow.
_ = append(merged, follow...)
}

684
vendor/github.com/boltdb/bolt/tx.go generated vendored
View File

@ -1,684 +0,0 @@
package bolt
import (
"fmt"
"io"
"os"
"sort"
"strings"
"time"
"unsafe"
)
// txid represents the internal transaction identifier.
type txid uint64
// Tx represents a read-only or read/write transaction on the database.
// Read-only transactions can be used for retrieving values for keys and creating cursors.
// Read/write transactions can create and remove buckets and create and remove keys.
//
// IMPORTANT: You must commit or rollback transactions when you are done with
// them. Pages can not be reclaimed by the writer until no more transactions
// are using them. A long running read transaction can cause the database to
// quickly grow.
type Tx struct {
writable bool
managed bool
db *DB
meta *meta
root Bucket
pages map[pgid]*page
stats TxStats
commitHandlers []func()
// WriteFlag specifies the flag for write-related methods like WriteTo().
// Tx opens the database file with the specified flag to copy the data.
//
// By default, the flag is unset, which works well for mostly in-memory
// workloads. For databases that are much larger than available RAM,
// set the flag to syscall.O_DIRECT to avoid trashing the page cache.
WriteFlag int
}
// init initializes the transaction.
func (tx *Tx) init(db *DB) {
tx.db = db
tx.pages = nil
// Copy the meta page since it can be changed by the writer.
tx.meta = &meta{}
db.meta().copy(tx.meta)
// Copy over the root bucket.
tx.root = newBucket(tx)
tx.root.bucket = &bucket{}
*tx.root.bucket = tx.meta.root
// Increment the transaction id and add a page cache for writable transactions.
if tx.writable {
tx.pages = make(map[pgid]*page)
tx.meta.txid += txid(1)
}
}
// ID returns the transaction id.
func (tx *Tx) ID() int {
return int(tx.meta.txid)
}
// DB returns a reference to the database that created the transaction.
func (tx *Tx) DB() *DB {
return tx.db
}
// Size returns current database size in bytes as seen by this transaction.
func (tx *Tx) Size() int64 {
return int64(tx.meta.pgid) * int64(tx.db.pageSize)
}
// Writable returns whether the transaction can perform write operations.
func (tx *Tx) Writable() bool {
return tx.writable
}
// Cursor creates a cursor associated with the root bucket.
// All items in the cursor will return a nil value because all root bucket keys point to buckets.
// The cursor is only valid as long as the transaction is open.
// Do not use a cursor after the transaction is closed.
func (tx *Tx) Cursor() *Cursor {
return tx.root.Cursor()
}
// Stats retrieves a copy of the current transaction statistics.
func (tx *Tx) Stats() TxStats {
return tx.stats
}
// Bucket retrieves a bucket by name.
// Returns nil if the bucket does not exist.
// The bucket instance is only valid for the lifetime of the transaction.
func (tx *Tx) Bucket(name []byte) *Bucket {
return tx.root.Bucket(name)
}
// CreateBucket creates a new bucket.
// Returns an error if the bucket already exists, if the bucket name is blank, or if the bucket name is too long.
// The bucket instance is only valid for the lifetime of the transaction.
func (tx *Tx) CreateBucket(name []byte) (*Bucket, error) {
return tx.root.CreateBucket(name)
}
// CreateBucketIfNotExists creates a new bucket if it doesn't already exist.
// Returns an error if the bucket name is blank, or if the bucket name is too long.
// The bucket instance is only valid for the lifetime of the transaction.
func (tx *Tx) CreateBucketIfNotExists(name []byte) (*Bucket, error) {
return tx.root.CreateBucketIfNotExists(name)
}
// DeleteBucket deletes a bucket.
// Returns an error if the bucket cannot be found or if the key represents a non-bucket value.
func (tx *Tx) DeleteBucket(name []byte) error {
return tx.root.DeleteBucket(name)
}
// ForEach executes a function for each bucket in the root.
// If the provided function returns an error then the iteration is stopped and
// the error is returned to the caller.
func (tx *Tx) ForEach(fn func(name []byte, b *Bucket) error) error {
return tx.root.ForEach(func(k, v []byte) error {
if err := fn(k, tx.root.Bucket(k)); err != nil {
return err
}
return nil
})
}
// OnCommit adds a handler function to be executed after the transaction successfully commits.
func (tx *Tx) OnCommit(fn func()) {
tx.commitHandlers = append(tx.commitHandlers, fn)
}
// Commit writes all changes to disk and updates the meta page.
// Returns an error if a disk write error occurs, or if Commit is
// called on a read-only transaction.
func (tx *Tx) Commit() error {
_assert(!tx.managed, "managed tx commit not allowed")
if tx.db == nil {
return ErrTxClosed
} else if !tx.writable {
return ErrTxNotWritable
}
// TODO(benbjohnson): Use vectorized I/O to write out dirty pages.
// Rebalance nodes which have had deletions.
var startTime = time.Now()
tx.root.rebalance()
if tx.stats.Rebalance > 0 {
tx.stats.RebalanceTime += time.Since(startTime)
}
// spill data onto dirty pages.
startTime = time.Now()
if err := tx.root.spill(); err != nil {
tx.rollback()
return err
}
tx.stats.SpillTime += time.Since(startTime)
// Free the old root bucket.
tx.meta.root.root = tx.root.root
opgid := tx.meta.pgid
// Free the freelist and allocate new pages for it. This will overestimate
// the size of the freelist but not underestimate the size (which would be bad).
tx.db.freelist.free(tx.meta.txid, tx.db.page(tx.meta.freelist))
p, err := tx.allocate((tx.db.freelist.size() / tx.db.pageSize) + 1)
if err != nil {
tx.rollback()
return err
}
if err := tx.db.freelist.write(p); err != nil {
tx.rollback()
return err
}
tx.meta.freelist = p.id
// If the high water mark has moved up then attempt to grow the database.
if tx.meta.pgid > opgid {
if err := tx.db.grow(int(tx.meta.pgid+1) * tx.db.pageSize); err != nil {
tx.rollback()
return err
}
}
// Write dirty pages to disk.
startTime = time.Now()
if err := tx.write(); err != nil {
tx.rollback()
return err
}
// If strict mode is enabled then perform a consistency check.
// Only the first consistency error is reported in the panic.
if tx.db.StrictMode {
ch := tx.Check()
var errs []string
for {
err, ok := <-ch
if !ok {
break
}
errs = append(errs, err.Error())
}
if len(errs) > 0 {
panic("check fail: " + strings.Join(errs, "\n"))
}
}
// Write meta to disk.
if err := tx.writeMeta(); err != nil {
tx.rollback()
return err
}
tx.stats.WriteTime += time.Since(startTime)
// Finalize the transaction.
tx.close()
// Execute commit handlers now that the locks have been removed.
for _, fn := range tx.commitHandlers {
fn()
}
return nil
}
// Rollback closes the transaction and ignores all previous updates. Read-only
// transactions must be rolled back and not committed.
func (tx *Tx) Rollback() error {
_assert(!tx.managed, "managed tx rollback not allowed")
if tx.db == nil {
return ErrTxClosed
}
tx.rollback()
return nil
}
func (tx *Tx) rollback() {
if tx.db == nil {
return
}
if tx.writable {
tx.db.freelist.rollback(tx.meta.txid)
tx.db.freelist.reload(tx.db.page(tx.db.meta().freelist))
}
tx.close()
}
func (tx *Tx) close() {
if tx.db == nil {
return
}
if tx.writable {
// Grab freelist stats.
var freelistFreeN = tx.db.freelist.free_count()
var freelistPendingN = tx.db.freelist.pending_count()
var freelistAlloc = tx.db.freelist.size()
// Remove transaction ref & writer lock.
tx.db.rwtx = nil
tx.db.rwlock.Unlock()
// Merge statistics.
tx.db.statlock.Lock()
tx.db.stats.FreePageN = freelistFreeN
tx.db.stats.PendingPageN = freelistPendingN
tx.db.stats.FreeAlloc = (freelistFreeN + freelistPendingN) * tx.db.pageSize
tx.db.stats.FreelistInuse = freelistAlloc
tx.db.stats.TxStats.add(&tx.stats)
tx.db.statlock.Unlock()
} else {
tx.db.removeTx(tx)
}
// Clear all references.
tx.db = nil
tx.meta = nil
tx.root = Bucket{tx: tx}
tx.pages = nil
}
// Copy writes the entire database to a writer.
// This function exists for backwards compatibility. Use WriteTo() instead.
func (tx *Tx) Copy(w io.Writer) error {
_, err := tx.WriteTo(w)
return err
}
// WriteTo writes the entire database to a writer.
// If err == nil then exactly tx.Size() bytes will be written into the writer.
func (tx *Tx) WriteTo(w io.Writer) (n int64, err error) {
// Attempt to open reader with WriteFlag
f, err := os.OpenFile(tx.db.path, os.O_RDONLY|tx.WriteFlag, 0)
if err != nil {
return 0, err
}
defer func() { _ = f.Close() }()
// Generate a meta page. We use the same page data for both meta pages.
buf := make([]byte, tx.db.pageSize)
page := (*page)(unsafe.Pointer(&buf[0]))
page.flags = metaPageFlag
*page.meta() = *tx.meta
// Write meta 0.
page.id = 0
page.meta().checksum = page.meta().sum64()
nn, err := w.Write(buf)
n += int64(nn)
if err != nil {
return n, fmt.Errorf("meta 0 copy: %s", err)
}
// Write meta 1 with a lower transaction id.
page.id = 1
page.meta().txid -= 1
page.meta().checksum = page.meta().sum64()
nn, err = w.Write(buf)
n += int64(nn)
if err != nil {
return n, fmt.Errorf("meta 1 copy: %s", err)
}
// Move past the meta pages in the file.
if _, err := f.Seek(int64(tx.db.pageSize*2), os.SEEK_SET); err != nil {
return n, fmt.Errorf("seek: %s", err)
}
// Copy data pages.
wn, err := io.CopyN(w, f, tx.Size()-int64(tx.db.pageSize*2))
n += wn
if err != nil {
return n, err
}
return n, f.Close()
}
// CopyFile copies the entire database to file at the given path.
// A reader transaction is maintained during the copy so it is safe to continue
// using the database while a copy is in progress.
func (tx *Tx) CopyFile(path string, mode os.FileMode) error {
f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, mode)
if err != nil {
return err
}
err = tx.Copy(f)
if err != nil {
_ = f.Close()
return err
}
return f.Close()
}
// Check performs several consistency checks on the database for this transaction.
// An error is returned if any inconsistency is found.
//
// It can be safely run concurrently on a writable transaction. However, this
// incurs a high cost for large databases and databases with a lot of subbuckets
// because of caching. This overhead can be removed if running on a read-only
// transaction, however, it is not safe to execute other writer transactions at
// the same time.
func (tx *Tx) Check() <-chan error {
ch := make(chan error)
go tx.check(ch)
return ch
}
func (tx *Tx) check(ch chan error) {
// Check if any pages are double freed.
freed := make(map[pgid]bool)
all := make([]pgid, tx.db.freelist.count())
tx.db.freelist.copyall(all)
for _, id := range all {
if freed[id] {
ch <- fmt.Errorf("page %d: already freed", id)
}
freed[id] = true
}
// Track every reachable page.
reachable := make(map[pgid]*page)
reachable[0] = tx.page(0) // meta0
reachable[1] = tx.page(1) // meta1
for i := uint32(0); i <= tx.page(tx.meta.freelist).overflow; i++ {
reachable[tx.meta.freelist+pgid(i)] = tx.page(tx.meta.freelist)
}
// Recursively check buckets.
tx.checkBucket(&tx.root, reachable, freed, ch)
// Ensure all pages below high water mark are either reachable or freed.
for i := pgid(0); i < tx.meta.pgid; i++ {
_, isReachable := reachable[i]
if !isReachable && !freed[i] {
ch <- fmt.Errorf("page %d: unreachable unfreed", int(i))
}
}
// Close the channel to signal completion.
close(ch)
}
func (tx *Tx) checkBucket(b *Bucket, reachable map[pgid]*page, freed map[pgid]bool, ch chan error) {
// Ignore inline buckets.
if b.root == 0 {
return
}
// Check every page used by this bucket.
b.tx.forEachPage(b.root, 0, func(p *page, _ int) {
if p.id > tx.meta.pgid {
ch <- fmt.Errorf("page %d: out of bounds: %d", int(p.id), int(b.tx.meta.pgid))
}
// Ensure each page is only referenced once.
for i := pgid(0); i <= pgid(p.overflow); i++ {
var id = p.id + i
if _, ok := reachable[id]; ok {
ch <- fmt.Errorf("page %d: multiple references", int(id))
}
reachable[id] = p
}
// We should only encounter un-freed leaf and branch pages.
if freed[p.id] {
ch <- fmt.Errorf("page %d: reachable freed", int(p.id))
} else if (p.flags&branchPageFlag) == 0 && (p.flags&leafPageFlag) == 0 {
ch <- fmt.Errorf("page %d: invalid type: %s", int(p.id), p.typ())
}
})
// Check each bucket within this bucket.
_ = b.ForEach(func(k, v []byte) error {
if child := b.Bucket(k); child != nil {
tx.checkBucket(child, reachable, freed, ch)
}
return nil
})
}
// allocate returns a contiguous block of memory starting at a given page.
func (tx *Tx) allocate(count int) (*page, error) {
p, err := tx.db.allocate(count)
if err != nil {
return nil, err
}
// Save to our page cache.
tx.pages[p.id] = p
// Update statistics.
tx.stats.PageCount++
tx.stats.PageAlloc += count * tx.db.pageSize
return p, nil
}
// write writes any dirty pages to disk.
func (tx *Tx) write() error {
// Sort pages by id.
pages := make(pages, 0, len(tx.pages))
for _, p := range tx.pages {
pages = append(pages, p)
}
// Clear out page cache early.
tx.pages = make(map[pgid]*page)
sort.Sort(pages)
// Write pages to disk in order.
for _, p := range pages {
size := (int(p.overflow) + 1) * tx.db.pageSize
offset := int64(p.id) * int64(tx.db.pageSize)
// Write out page in "max allocation" sized chunks.
ptr := (*[maxAllocSize]byte)(unsafe.Pointer(p))
for {
// Limit our write to our max allocation size.
sz := size
if sz > maxAllocSize-1 {
sz = maxAllocSize - 1
}
// Write chunk to disk.
buf := ptr[:sz]
if _, err := tx.db.ops.writeAt(buf, offset); err != nil {
return err
}
// Update statistics.
tx.stats.Write++
// Exit inner for loop if we've written all the chunks.
size -= sz
if size == 0 {
break
}
// Otherwise move offset forward and move pointer to next chunk.
offset += int64(sz)
ptr = (*[maxAllocSize]byte)(unsafe.Pointer(&ptr[sz]))
}
}
// Ignore file sync if flag is set on DB.
if !tx.db.NoSync || IgnoreNoSync {
if err := fdatasync(tx.db); err != nil {
return err
}
}
// Put small pages back to page pool.
for _, p := range pages {
// Ignore page sizes over 1 page.
// These are allocated using make() instead of the page pool.
if int(p.overflow) != 0 {
continue
}
buf := (*[maxAllocSize]byte)(unsafe.Pointer(p))[:tx.db.pageSize]
// See https://go.googlesource.com/go/+/f03c9202c43e0abb130669852082117ca50aa9b1
for i := range buf {
buf[i] = 0
}
tx.db.pagePool.Put(buf)
}
return nil
}
// writeMeta writes the meta to the disk.
func (tx *Tx) writeMeta() error {
// Create a temporary buffer for the meta page.
buf := make([]byte, tx.db.pageSize)
p := tx.db.pageInBuffer(buf, 0)
tx.meta.write(p)
// Write the meta page to file.
if _, err := tx.db.ops.writeAt(buf, int64(p.id)*int64(tx.db.pageSize)); err != nil {
return err
}
if !tx.db.NoSync || IgnoreNoSync {
if err := fdatasync(tx.db); err != nil {
return err
}
}
// Update statistics.
tx.stats.Write++
return nil
}
// page returns a reference to the page with a given id.
// If page has been written to then a temporary buffered page is returned.
func (tx *Tx) page(id pgid) *page {
// Check the dirty pages first.
if tx.pages != nil {
if p, ok := tx.pages[id]; ok {
return p
}
}
// Otherwise return directly from the mmap.
return tx.db.page(id)
}
// forEachPage iterates over every page within a given page and executes a function.
func (tx *Tx) forEachPage(pgid pgid, depth int, fn func(*page, int)) {
p := tx.page(pgid)
// Execute function.
fn(p, depth)
// Recursively loop over children.
if (p.flags & branchPageFlag) != 0 {
for i := 0; i < int(p.count); i++ {
elem := p.branchPageElement(uint16(i))
tx.forEachPage(elem.pgid, depth+1, fn)
}
}
}
// Page returns page information for a given page number.
// This is only safe for concurrent use when used by a writable transaction.
func (tx *Tx) Page(id int) (*PageInfo, error) {
if tx.db == nil {
return nil, ErrTxClosed
} else if pgid(id) >= tx.meta.pgid {
return nil, nil
}
// Build the page info.
p := tx.db.page(pgid(id))
info := &PageInfo{
ID: id,
Count: int(p.count),
OverflowCount: int(p.overflow),
}
// Determine the type (or if it's free).
if tx.db.freelist.freed(pgid(id)) {
info.Type = "free"
} else {
info.Type = p.typ()
}
return info, nil
}
// TxStats represents statistics about the actions performed by the transaction.
type TxStats struct {
// Page statistics.
PageCount int // number of page allocations
PageAlloc int // total bytes allocated
// Cursor statistics.
CursorCount int // number of cursors created
// Node statistics
NodeCount int // number of node allocations
NodeDeref int // number of node dereferences
// Rebalance statistics.
Rebalance int // number of node rebalances
RebalanceTime time.Duration // total time spent rebalancing
// Split/Spill statistics.
Split int // number of nodes split
Spill int // number of nodes spilled
SpillTime time.Duration // total time spent spilling
// Write statistics.
Write int // number of writes performed
WriteTime time.Duration // total time spent writing to disk
}
func (s *TxStats) add(other *TxStats) {
s.PageCount += other.PageCount
s.PageAlloc += other.PageAlloc
s.CursorCount += other.CursorCount
s.NodeCount += other.NodeCount
s.NodeDeref += other.NodeDeref
s.Rebalance += other.Rebalance
s.RebalanceTime += other.RebalanceTime
s.Split += other.Split
s.Spill += other.Spill
s.SpillTime += other.SpillTime
s.Write += other.Write
s.WriteTime += other.WriteTime
}
// Sub calculates and returns the difference between two sets of transaction stats.
// This is useful when obtaining stats at two different points and time and
// you need the performance counters that occurred within that time span.
func (s *TxStats) Sub(other *TxStats) TxStats {
var diff TxStats
diff.PageCount = s.PageCount - other.PageCount
diff.PageAlloc = s.PageAlloc - other.PageAlloc
diff.CursorCount = s.CursorCount - other.CursorCount
diff.NodeCount = s.NodeCount - other.NodeCount
diff.NodeDeref = s.NodeDeref - other.NodeDeref
diff.Rebalance = s.Rebalance - other.Rebalance
diff.RebalanceTime = s.RebalanceTime - other.RebalanceTime
diff.Split = s.Split - other.Split
diff.Spill = s.Spill - other.Spill
diff.SpillTime = s.SpillTime - other.SpillTime
diff.Write = s.Write - other.Write
diff.WriteTime = s.WriteTime - other.WriteTime
return diff
}

15
vendor/github.com/cznic/fileutil/AUTHORS generated vendored
View File

@ -1,15 +0,0 @@
# This file lists authors for copyright purposes. This file is distinct from
# the CONTRIBUTORS files. See the latter for an explanation.
#
# Names should be added to this file as:
# Name or Organization <email address>
#
# The email address is not required for organizations.
#
# Please keep the list sorted.
CZ.NIC z.s.p.o. <kontakt@nic.cz>
Jan Mercl <0xjnml@gmail.com>
Linelane GmbH <info@linelane.com>
Aaron Bieber <deftly@gmail.com>

15
vendor/github.com/cznic/fileutil/CONTRIBUTORS generated vendored
View File

@ -1,15 +0,0 @@
# This file lists people who contributed code to this repository. The AUTHORS
# file lists the copyright holders; this file lists people.
#
# Names should be added to this file like so:
# Name <email address>
#
# Please keep the list sorted.
Andris Valums <info@linelane.com>
Bill Thiede <xinu.tv>
Gary Burd <gary@beagledreams.com>
Jan Mercl <0xjnml@gmail.com>
Nick Owens <mischief@offblast.org>
Tamás Gulácsi <gt-dev@gthomas.eu>
Aaron Bieber <deftly@gmail.com>

27
vendor/github.com/cznic/fileutil/LICENSE generated vendored
View File

@ -1,27 +0,0 @@
Copyright (c) 2014 The fileutil Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the names of the authors nor the names of the
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

27
vendor/github.com/cznic/fileutil/Makefile generated vendored
View File

@ -1,27 +0,0 @@
# Copyright (c) 2014 The fileutil authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.
.PHONY: all clean editor todo
all: editor
go vet
golint .
go install
make todo
editor:
go fmt
go test -i
go test
go build
todo:
@grep -n ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* *.go || true
@grep -n TODO *.go || true
@grep -n BUG *.go || true
@grep -n println *.go || true
clean:
@go clean
rm -f y.output

16
vendor/github.com/cznic/fileutil/README generated vendored
View File

@ -1,16 +0,0 @@
This is a goinstall-able mirror of modified code already published at:
http://git.nic.cz/redmine/projects/gofileutil/repository
Packages in this repository:
Install: $go get github.com/cznic/fileutil
Godocs: http://godoc.org/github.com/cznic/fileutil
Install: $go get github.com/cznic/fileutil/storage
Godocs: http://godoc.org/github.com/cznic/fileutil/storage
Install: $go get github.com/cznic/fileutil/falloc
Godocs: http://godoc.org/github.com/cznic/fileutil/falloc
Install: $go get github.com/cznic/fileutil/hdb
Godocs: http://godoc.org/github.com/cznic/fileutil/hdb

223
vendor/github.com/cznic/fileutil/fileutil.go generated vendored
View File

@ -1,223 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package fileutil collects some file utility functions.
package fileutil
import (
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"strconv"
"sync"
"time"
)
// GoMFile is a concurrent access safe version of MFile.
type GoMFile struct {
mfile *MFile
mutex sync.Mutex
}
// NewGoMFile return a newly created GoMFile.
func NewGoMFile(fname string, flag int, perm os.FileMode, delta_ns int64) (m *GoMFile, err error) {
m = &GoMFile{}
if m.mfile, err = NewMFile(fname, flag, perm, delta_ns); err != nil {
m = nil
}
return
}
func (m *GoMFile) File() (file *os.File, err error) {
m.mutex.Lock()
defer m.mutex.Unlock()
return m.mfile.File()
}
func (m *GoMFile) SetChanged() {
m.mutex.Lock()
defer m.mutex.Unlock()
m.mfile.SetChanged()
}
func (m *GoMFile) SetHandler(h MFileHandler) {
m.mutex.Lock()
defer m.mutex.Unlock()
m.mfile.SetHandler(h)
}
// MFileHandler resolves modifications of File.
// Possible File context is expected to be a part of the handler's closure.
type MFileHandler func(*os.File) error
// MFile represents an os.File with a guard/handler on change/modification.
// Example use case is an app with a configuration file which can be modified at any time
// and have to be reloaded in such event prior to performing something configurable by that
// file. The checks are made only on access to the MFile file by
// File() and a time threshold/hysteresis value can be chosen on creating a new MFile.
type MFile struct {
file *os.File
handler MFileHandler
t0 int64
delta int64
ctime int64
}
// NewMFile returns a newly created MFile or Error if any.
// The fname, flag and perm parameters have the same meaning as in os.Open.
// For meaning of the delta_ns parameter please see the (m *MFile) File() docs.
func NewMFile(fname string, flag int, perm os.FileMode, delta_ns int64) (m *MFile, err error) {
m = &MFile{}
m.t0 = time.Now().UnixNano()
if m.file, err = os.OpenFile(fname, flag, perm); err != nil {
return
}
var fi os.FileInfo
if fi, err = m.file.Stat(); err != nil {
return
}
m.ctime = fi.ModTime().UnixNano()
m.delta = delta_ns
runtime.SetFinalizer(m, func(m *MFile) {
m.file.Close()
})
return
}
// SetChanged forces next File() to unconditionally handle modification of the wrapped os.File.
func (m *MFile) SetChanged() {
m.ctime = -1
}
// SetHandler sets a function to be invoked when modification of MFile is to be processed.
func (m *MFile) SetHandler(h MFileHandler) {
m.handler = h
}
// File returns an os.File from MFile. If time elapsed between the last invocation of this function
// and now is at least delta_ns ns (a parameter of NewMFile) then the file is checked for
// change/modification. For delta_ns == 0 the modification is checked w/o getting os.Time().
// If a change is detected a handler is invoked on the MFile file.
// Any of these steps can produce an Error. If that happens the function returns nil, Error.
func (m *MFile) File() (file *os.File, err error) {
var now int64
mustCheck := m.delta == 0
if !mustCheck {
now = time.Now().UnixNano()
mustCheck = now-m.t0 > m.delta
}
if mustCheck { // check interval reached
var fi os.FileInfo
if fi, err = m.file.Stat(); err != nil {
return
}
if fi.ModTime().UnixNano() != m.ctime { // modification detected
if m.handler == nil {
return nil, fmt.Errorf("no handler set for modified file %q", m.file.Name())
}
if err = m.handler(m.file); err != nil {
return
}
m.ctime = fi.ModTime().UnixNano()
}
m.t0 = now
}
return m.file, nil
}
// Read reads buf from r. It will either fill the full buf or fail.
// It wraps the functionality of an io.Reader which may return less bytes than requested,
// but may block if not all data are ready for the io.Reader.
func Read(r io.Reader, buf []byte) (err error) {
have := 0
remain := len(buf)
got := 0
for remain > 0 {
if got, err = r.Read(buf[have:]); err != nil {
return
}
remain -= got
have += got
}
return
}
// "os" and/or "syscall" extensions
// FadviseAdvice is used by Fadvise.
type FadviseAdvice int
// FAdviseAdvice values.
const (
// $ grep FADV /usr/include/bits/fcntl.h
POSIX_FADV_NORMAL FadviseAdvice = iota // No further special treatment.
POSIX_FADV_RANDOM // Expect random page references.
POSIX_FADV_SEQUENTIAL // Expect sequential page references.
POSIX_FADV_WILLNEED // Will need these pages.
POSIX_FADV_DONTNEED // Don't need these pages.
POSIX_FADV_NOREUSE // Data will be accessed once.
)
// TempFile creates a new temporary file in the directory dir with a name
// ending with suffix, basename starting with prefix, opens the file for
// reading and writing, and returns the resulting *os.File. If dir is the
// empty string, TempFile uses the default directory for temporary files (see
// os.TempDir). Multiple programs calling TempFile simultaneously will not
// choose the same file. The caller can use f.Name() to find the pathname of
// the file. It is the caller's responsibility to remove the file when no
// longer needed.
//
// NOTE: This function differs from ioutil.TempFile.
func TempFile(dir, prefix, suffix string) (f *os.File, err error) {
if dir == "" {
dir = os.TempDir()
}
nconflict := 0
for i := 0; i < 10000; i++ {
name := filepath.Join(dir, prefix+nextInfix()+suffix)
f, err = os.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0600)
if os.IsExist(err) {
if nconflict++; nconflict > 10 {
rand = reseed()
}
continue
}
break
}
return
}
// Random number state.
// We generate random temporary file names so that there's a good
// chance the file doesn't exist yet - keeps the number of tries in
// TempFile to a minimum.
var rand uint32
var randmu sync.Mutex
func reseed() uint32 {
return uint32(time.Now().UnixNano() + int64(os.Getpid()))
}
func nextInfix() string {
randmu.Lock()
r := rand
if r == 0 {
r = reseed()
}
r = r*1664525 + 1013904223 // constants from Numerical Recipes
rand = r
randmu.Unlock()
return strconv.Itoa(int(1e9 + r%1e9))[1:]
}

27
vendor/github.com/cznic/fileutil/fileutil_arm.go generated vendored
View File

@ -1,27 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
)
const hasPunchHole = false
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Not supported on ARM.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Not supported on ARM.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

29
vendor/github.com/cznic/fileutil/fileutil_darwin.go generated vendored
View File

@ -1,29 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !arm
package fileutil
import (
"io"
"os"
)
const hasPunchHole = false
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Not supported on OSX.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Not supported on OSX.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

29
vendor/github.com/cznic/fileutil/fileutil_freebsd.go generated vendored
View File

@ -1,29 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !arm
package fileutil
import (
"io"
"os"
)
const hasPunchHole = false
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Unimplemented on FreeBSD.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Unimplemented on FreeBSD.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

98
vendor/github.com/cznic/fileutil/fileutil_linux.go generated vendored
View File

@ -1,98 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !arm
package fileutil
import (
"bytes"
"io"
"io/ioutil"
"os"
"strconv"
"syscall"
)
const hasPunchHole = true
func n(s []byte) byte {
for i, c := range s {
if c < '0' || c > '9' {
s = s[:i]
break
}
}
v, _ := strconv.Atoi(string(s))
return byte(v)
}
func init() {
b, err := ioutil.ReadFile("/proc/sys/kernel/osrelease")
if err != nil {
panic(err)
}
tokens := bytes.Split(b, []byte("."))
if len(tokens) > 3 {
tokens = tokens[:3]
}
switch len(tokens) {
case 3:
// Supported since kernel 2.6.38
if bytes.Compare([]byte{n(tokens[0]), n(tokens[1]), n(tokens[2])}, []byte{2, 6, 38}) < 0 {
puncher = func(*os.File, int64, int64) error { return nil }
}
case 2:
if bytes.Compare([]byte{n(tokens[0]), n(tokens[1])}, []byte{2, 7}) < 0 {
puncher = func(*os.File, int64, int64) error { return nil }
}
default:
puncher = func(*os.File, int64, int64) error { return nil }
}
}
var puncher = func(f *os.File, off, len int64) error {
const (
/*
/usr/include/linux$ grep FL_ falloc.h
*/
_FALLOC_FL_KEEP_SIZE = 0x01 // default is extend size
_FALLOC_FL_PUNCH_HOLE = 0x02 // de-allocates range
)
_, _, errno := syscall.Syscall6(
syscall.SYS_FALLOCATE,
uintptr(f.Fd()),
uintptr(_FALLOC_FL_KEEP_SIZE|_FALLOC_FL_PUNCH_HOLE),
uintptr(off),
uintptr(len),
0, 0)
if errno != 0 {
return os.NewSyscallError("SYS_FALLOCATE", errno)
}
return nil
}
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. No-op for kernels < 2.6.38 (or < 2.7).
func PunchHole(f *os.File, off, len int64) error {
return puncher(f, off, len)
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
_, _, errno := syscall.Syscall6(
syscall.SYS_FADVISE64,
uintptr(f.Fd()),
uintptr(off),
uintptr(len),
uintptr(advice),
0, 0)
return os.NewSyscallError("SYS_FADVISE64", errno)
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

29
vendor/github.com/cznic/fileutil/fileutil_netbsd.go generated vendored
View File

@ -1,29 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build !arm
package fileutil
import (
"io"
"os"
)
const hasPunchHole = false
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Similar to FreeBSD, this is
// unimplemented.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Unimplemented on NetBSD.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

27
vendor/github.com/cznic/fileutil/fileutil_openbsd.go generated vendored
View File

@ -1,27 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
)
const hasPunchHole = false
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Similar to FreeBSD, this is
// unimplemented.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Unimplemented on OpenBSD.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

27
vendor/github.com/cznic/fileutil/fileutil_plan9.go generated vendored
View File

@ -1,27 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
)
const hasPunchHole = false
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Unimplemented on Plan 9.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Unimplemented on Plan 9.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

29
vendor/github.com/cznic/fileutil/fileutil_solaris.go generated vendored
View File

@ -1,29 +0,0 @@
// Copyright (c) 2013 jnml. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build go1.3
package fileutil
import (
"io"
"os"
)
const hasPunchHole = false
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Not supported on Solaris.
func PunchHole(f *os.File, off, len int64) error {
return nil
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Not supported on Solaris.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool { return err == io.EOF }

185
vendor/github.com/cznic/fileutil/fileutil_windows.go generated vendored
View File

@ -1,185 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"io"
"os"
"sync"
"syscall"
"unsafe"
)
const hasPunchHole = true
// PunchHole deallocates space inside a file in the byte range starting at
// offset and continuing for len bytes. Not supported on Windows.
func PunchHole(f *os.File, off, len int64) error {
return puncher(f, off, len)
}
// Fadvise predeclares an access pattern for file data. See also 'man 2
// posix_fadvise'. Not supported on Windows.
func Fadvise(f *os.File, off, len int64, advice FadviseAdvice) error {
return nil
}
// IsEOF reports whether err is an EOF condition.
func IsEOF(err error) bool {
if err == io.EOF {
return true
}
// http://social.technet.microsoft.com/Forums/windowsserver/en-US/1a16311b-c625-46cf-830b-6a26af488435/how-to-solve-error-38-0x26-errorhandleeof-using-fsctlgetretrievalpointers
x, ok := err.(*os.PathError)
return ok && x.Op == "read" && x.Err.(syscall.Errno) == 0x26
}
var (
modkernel32 = syscall.NewLazyDLL("kernel32.dll")
procDeviceIOControl = modkernel32.NewProc("DeviceIoControl")
sparseFilesMu sync.Mutex
sparseFiles map[uintptr]struct{}
)
func init() {
// sparseFiles is an fd set for already "sparsed" files - according to
// msdn.microsoft.com/en-us/library/windows/desktop/aa364225(v=vs.85).aspx
// the file handles are unique per process.
sparseFiles = make(map[uintptr]struct{})
}
// puncHoleWindows punches a hole into the given file starting at offset,
// measuring "size" bytes
// (http://msdn.microsoft.com/en-us/library/windows/desktop/aa364597%28v=vs.85%29.aspx)
func puncher(file *os.File, offset, size int64) error {
if err := ensureFileSparse(file); err != nil {
return err
}
// http://msdn.microsoft.com/en-us/library/windows/desktop/aa364411%28v=vs.85%29.aspx
// typedef struct _FILE_ZERO_DATA_INFORMATION {
// LARGE_INTEGER FileOffset;
// LARGE_INTEGER BeyondFinalZero;
//} FILE_ZERO_DATA_INFORMATION, *PFILE_ZERO_DATA_INFORMATION;
type fileZeroDataInformation struct {
FileOffset, BeyondFinalZero int64
}
lpInBuffer := fileZeroDataInformation{
FileOffset: offset,
BeyondFinalZero: offset + size}
return deviceIOControl(false, file.Fd(), uintptr(unsafe.Pointer(&lpInBuffer)), 16)
}
// // http://msdn.microsoft.com/en-us/library/windows/desktop/cc948908%28v=vs.85%29.aspx
// type fileSetSparseBuffer struct {
// SetSparse bool
// }
func ensureFileSparse(file *os.File) (err error) {
fd := file.Fd()
sparseFilesMu.Lock()
if _, ok := sparseFiles[fd]; ok {
sparseFilesMu.Unlock()
return nil
}
if err = deviceIOControl(true, fd, 0, 0); err == nil {
sparseFiles[fd] = struct{}{}
}
sparseFilesMu.Unlock()
return err
}
func deviceIOControl(setSparse bool, fd, inBuf, inBufLen uintptr) (err error) {
const (
//http://source.winehq.org/source/include/winnt.h#L4605
file_read_data = 1
file_write_data = 2
// METHOD_BUFFERED 0
method_buffered = 0
// FILE_ANY_ACCESS 0
file_any_access = 0
// FILE_DEVICE_FILE_SYSTEM 0x00000009
file_device_file_system = 0x00000009
// FILE_SPECIAL_ACCESS (FILE_ANY_ACCESS)
file_special_access = file_any_access
file_read_access = file_read_data
file_write_access = file_write_data
// http://source.winehq.org/source/include/winioctl.h
// #define CTL_CODE ( DeviceType,
// Function,
// Method,
// Access )
// ((DeviceType) << 16) | ((Access) << 14) | ((Function) << 2) | (Method)
// FSCTL_SET_COMPRESSION CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 16, METHOD_BUFFERED, FILE_READ_DATA | FILE_WRITE_DATA)
fsctl_set_compression = (file_device_file_system << 16) | ((file_read_access | file_write_access) << 14) | (16 << 2) | method_buffered
// FSCTL_SET_SPARSE CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 49, METHOD_BUFFERED, FILE_SPECIAL_ACCESS)
fsctl_set_sparse = (file_device_file_system << 16) | (file_special_access << 14) | (49 << 2) | method_buffered
// FSCTL_SET_ZERO_DATA CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 50, METHOD_BUFFERED, FILE_WRITE_DATA)
fsctl_set_zero_data = (file_device_file_system << 16) | (file_write_data << 14) | (50 << 2) | method_buffered
)
retPtr := uintptr(unsafe.Pointer(&(make([]byte, 8)[0])))
var r1 uintptr
var e1 syscall.Errno
if setSparse {
// BOOL
// WINAPI
// DeviceIoControl( (HANDLE) hDevice, // handle to a file
// FSCTL_SET_SPARSE, // dwIoControlCode
// (PFILE_SET_SPARSE_BUFFER) lpInBuffer, // input buffer
// (DWORD) nInBufferSize, // size of input buffer
// NULL, // lpOutBuffer
// 0, // nOutBufferSize
// (LPDWORD) lpBytesReturned, // number of bytes returned
// (LPOVERLAPPED) lpOverlapped ); // OVERLAPPED structure
r1, _, e1 = syscall.Syscall9(procDeviceIOControl.Addr(), 8,
fd,
uintptr(fsctl_set_sparse),
// If the lpInBuffer parameter is NULL, the operation will behave the same as if the SetSparse member of the FILE_SET_SPARSE_BUFFER structure were TRUE. In other words, the operation sets the file to a sparse file.
0, // uintptr(unsafe.Pointer(&lpInBuffer)),
0, // 1,
0,
0,
retPtr,
0,
0)
} else {
// BOOL
// WINAPI
// DeviceIoControl( (HANDLE) hDevice, // handle to a file
// FSCTL_SET_ZERO_DATA, // dwIoControlCode
// (LPVOID) lpInBuffer, // input buffer
// (DWORD) nInBufferSize, // size of input buffer
// NULL, // lpOutBuffer
// 0, // nOutBufferSize
// (LPDWORD) lpBytesReturned, // number of bytes returned
// (LPOVERLAPPED) lpOverlapped ); // OVERLAPPED structure
r1, _, e1 = syscall.Syscall9(procDeviceIOControl.Addr(), 8,
fd,
uintptr(fsctl_set_zero_data),
inBuf,
inBufLen,
0,
0,
retPtr,
0,
0)
}
if r1 == 0 {
if e1 != 0 {
err = error(e1)
} else {
err = syscall.EINVAL
}
}
return err
}

13
vendor/github.com/cznic/fileutil/test_deps.go generated vendored
View File

@ -1,13 +0,0 @@
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package fileutil
// Pull test dependencies too.
// Enables easy 'go test X' after 'go get X'
import (
// nothing yet
)

60
vendor/github.com/cznic/internal/buffer/Makefile generated vendored
View File

@ -1,60 +0,0 @@
# Copyright 2016 The Internal Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.
.PHONY: all clean cover cpu editor internalError later mem nuke todo edit
grep=--include=*.go --include=*.l --include=*.y --include=*.yy
ngrep='TODOOK\|parser\.go\|scanner\.go\|.*_string\.go'
all: editor
go vet 2>&1 | grep -v $(ngrep) || true
go vet 2>&1 | grep -v $(ngrep) || true
golint 2>&1 | grep -v $(ngrep) || true
make todo
unused . || true
misspell *.go
gosimple || true
codesweep || true
maligned || true
unconvert -apply
clean:
go clean
rm -f *~ *.test *.out
cover:
t=$(shell tempfile) ; go test -coverprofile $$t && go tool cover -html $$t && unlink $$t
cpu: clean
go test -run @ -bench . -cpuprofile cpu.out
go tool pprof -lines *.test cpu.out
edit:
@ 1>/dev/null 2>/dev/null gvim -p Makefile log *.go
editor:
gofmt -l -s -w *.go
go test -i
go test 2>&1 | tee log
go install
internalError:
egrep -ho '"internal error.*"' *.go | sort | cat -n
later:
@grep -n $(grep) LATER * || true
@grep -n $(grep) MAYBE * || true
mem: clean
go test -run @ -bench . -memprofile mem.out -memprofilerate 1 -timeout 24h
go tool pprof -lines -web -alloc_space *.test mem.out
nuke: clean
go clean -i
todo:
@grep -nr $(grep) ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* * | grep -v $(ngrep) || true
@grep -nr $(grep) TODO * | grep -v $(ngrep) || true
@grep -nr $(grep) BUG * | grep -v $(ngrep) || true
@grep -nr $(grep) [^[:alpha:]]println * | grep -v $(ngrep) || true

146
vendor/github.com/cznic/internal/buffer/buffer.go generated vendored
View File

@ -1,146 +0,0 @@
// Copyright 2016 The Internal Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package buffer implements a pool of pointers to byte slices.
//
// Example usage pattern
//
// p := buffer.Get(size)
// b := *p // Now you can use b in any way you need.
// ...
// // When b will not be used anymore
// buffer.Put(p)
// ...
// // If b or p are not going out of scope soon, optionally
// b = nil
// p = nil
//
// Otherwise the pool cannot release the buffer on garbage collection.
//
// Do not do
//
// p := buffer.Get(size)
// b := *p
// ...
// buffer.Put(&b)
//
// or
//
// b := *buffer.Get(size)
// ...
// buffer.Put(&b)
package buffer
import (
"github.com/cznic/internal/slice"
"io"
)
// CGet returns a pointer to a byte slice of len size. The pointed to byte
// slice is zeroed up to its cap. CGet panics for size < 0.
//
// CGet is safe for concurrent use by multiple goroutines.
func CGet(size int) *[]byte { return slice.Bytes.CGet(size).(*[]byte) }
// Get returns a pointer to a byte slice of len size. The pointed to byte slice
// is not zeroed. Get panics for size < 0.
//
// Get is safe for concurrent use by multiple goroutines.
func Get(size int) *[]byte { return slice.Bytes.Get(size).(*[]byte) }
// Put puts a pointer to a byte slice into a pool for possible later reuse by
// CGet or Get.
//
// Put is safe for concurrent use by multiple goroutines.
func Put(p *[]byte) { slice.Bytes.Put(p) }
// Bytes is similar to bytes.Buffer but may generate less garbage when properly
// Closed. Zero value is ready to use.
type Bytes struct {
p *[]byte
}
// Bytes return the content of b. The result is R/O.
func (b *Bytes) Bytes() []byte {
if b.p != nil {
return *b.p
}
return nil
}
// Close will recycle the underlying storage, if any. After Close, b is again
// the zero value.
func (b *Bytes) Close() error {
if b.p != nil {
Put(b.p)
b.p = nil
}
return nil
}
// Len returns the size of content in b.
func (b *Bytes) Len() int {
if b.p != nil {
return len(*b.p)
}
return 0
}
// Reset discard the content of Bytes while keeping the internal storage, if any.
func (b *Bytes) Reset() {
if b.p != nil {
*b.p = (*b.p)[:0]
}
}
// Write writes p into b and returns (len(p), nil).
func (b *Bytes) Write(p []byte) (int, error) {
n := b.Len()
b.grow(n + len(p))
copy((*b.p)[n:], p)
return len(p), nil
}
// WriteByte writes p into b and returns nil.
func (b *Bytes) WriteByte(p byte) error {
n := b.Len()
b.grow(n + 1)
(*b.p)[n] = p
return nil
}
// WriteTo writes b's content to w and returns the number of bytes written to w
// and an error, if any.
func (b *Bytes) WriteTo(w io.Writer) (int64, error) {
n, err := w.Write(b.Bytes())
return int64(n), err
}
// WriteString writes s to b and returns (len(s), nil).
func (b *Bytes) WriteString(s string) (int, error) {
n := b.Len()
b.grow(n + len(s))
copy((*b.p)[n:], s)
return len(s), nil
}
func (b *Bytes) grow(n int) {
if b.p != nil {
if n <= cap(*b.p) {
*b.p = (*b.p)[:n]
return
}
np := Get(2 * n)
*np = (*np)[:n]
copy(*np, *b.p)
Put(b.p)
b.p = np
return
}
b.p = Get(n)
}

55
vendor/github.com/cznic/internal/file/Makefile generated vendored
View File

@ -1,55 +0,0 @@
# Copyright 2016 The Internal Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.
.PHONY: all clean cover cpu editor internalError later mem nuke todo edit
grep=--include=*.go --include=*.l --include=*.y --include=*.yy
ngrep='TODOOK\|parser\.go\|scanner\.go\|.*_string\.go'
all: editor
go vet 2>&1 | grep -v $(ngrep) || true
golint 2>&1 | grep -v $(ngrep) || true
make todo
unused . || true
misspell *.go
gosimple || true
clean:
go clean
rm -f *~ *.test *.out
cover:
t=$(shell tempfile) ; go test -coverprofile $$t && go tool cover -html $$t && unlink $$t
cpu: clean
go test -run @ -bench . -cpuprofile cpu.out
go tool pprof -lines *.test cpu.out
edit:
@ 1>/dev/null 2>/dev/null gvim -p Makefile log *.go
editor:
gofmt -l -s -w *.go
go test 2>&1 | tee log
go build
internalError:
egrep -ho '"internal error.*"' *.go | sort | cat -n
later:
@grep -n $(grep) LATER * || true
@grep -n $(grep) MAYBE * || true
mem: clean
go test -run @ -bench BenchmarkReadWrite -benchmem -memprofile mem.out -memprofilerate 1 -timeout 24h
go tool pprof -lines -web -alloc_space *.test mem.out
nuke: clean
go clean -i
todo:
@grep -nr $(grep) ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* * | grep -v $(ngrep) || true
@grep -nr $(grep) TODO * | grep -v $(ngrep) || true
@grep -nr $(grep) BUG * | grep -v $(ngrep) || true
@grep -nr $(grep) [^[:alpha:]]println * | grep -v $(ngrep) || true

434
vendor/github.com/cznic/internal/file/file.go generated vendored
View File

@ -1,434 +0,0 @@
// Copyright 2016 The Internal Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package file provides an os.File-like interface of a memory mapped file.
package file
import (
"fmt"
"io"
"os"
"time"
"github.com/cznic/fileutil"
"github.com/cznic/internal/buffer"
"github.com/cznic/mathutil"
"github.com/edsrzf/mmap-go"
)
const copyBufSize = 1 << 20 // 1 MB.
var (
_ Interface = (*mem)(nil)
_ Interface = (*file)(nil)
_ os.FileInfo = stat{}
sysPage = os.Getpagesize()
)
// Interface is a os.File-like entity.
type Interface interface {
io.ReaderAt
io.ReaderFrom
io.WriterAt
io.WriterTo
Close() error
Stat() (os.FileInfo, error)
Sync() error
Truncate(int64) error
}
// Open returns a new Interface backed by f, or an error, if any.
func Open(f *os.File) (Interface, error) { return newFile(f, 1<<30, 20) }
// OpenMem returns a new Interface, or an error, if any. The Interface content
// is volatile, it's backed only by process' memory.
func OpenMem(name string) (Interface, error) { return newMem(name, 18), nil }
type memMap map[int64]*[]byte
type mem struct {
m memMap
modTime time.Time
name string
pgBits uint
pgMask int
pgSize int
size int64
}
func newMem(name string, pgBits uint) *mem {
pgSize := 1 << pgBits
return &mem{
m: memMap{},
modTime: time.Now(),
name: name,
pgBits: pgBits,
pgMask: pgSize - 1,
pgSize: pgSize,
}
}
func (f *mem) IsDir() bool { return false }
func (f *mem) Mode() os.FileMode { return os.ModeTemporary + 0600 }
func (f *mem) ModTime() time.Time { return f.modTime }
func (f *mem) Name() string { return f.name }
func (f *mem) ReadFrom(r io.Reader) (n int64, err error) { return readFrom(f, r) }
func (f *mem) Size() (n int64) { return f.size }
func (f *mem) Stat() (os.FileInfo, error) { return f, nil }
func (f *mem) Sync() error { return nil }
func (f *mem) Sys() interface{} { return nil }
func (f *mem) WriteTo(w io.Writer) (n int64, err error) { return writeTo(f, w) }
func (f *mem) Close() error {
f.Truncate(0)
f.m = nil
return nil
}
func (f *mem) ReadAt(b []byte, off int64) (n int, err error) {
avail := f.size - off
pi := off >> f.pgBits
po := int(off) & f.pgMask
rem := len(b)
if int64(rem) >= avail {
rem = int(avail)
err = io.EOF
}
var zeroPage *[]byte
for rem != 0 && avail > 0 {
pg := f.m[pi]
if pg == nil {
if zeroPage == nil {
zeroPage = buffer.CGet(f.pgSize)
defer buffer.Put(zeroPage)
}
pg = zeroPage
}
nc := copy(b[:mathutil.Min(rem, f.pgSize)], (*pg)[po:])
pi++
po = 0
rem -= nc
n += nc
b = b[nc:]
}
return n, err
}
func (f *mem) Truncate(size int64) (err error) {
if size < 0 {
return fmt.Errorf("invalid truncate size: %d", size)
}
first := size >> f.pgBits
if size&int64(f.pgMask) != 0 {
first++
}
last := f.size >> f.pgBits
if f.size&int64(f.pgMask) != 0 {
last++
}
for ; first <= last; first++ {
if p := f.m[first]; p != nil {
buffer.Put(p)
}
delete(f.m, first)
}
f.size = size
return nil
}
func (f *mem) WriteAt(b []byte, off int64) (n int, err error) {
pi := off >> f.pgBits
po := int(off) & f.pgMask
n = len(b)
rem := n
var nc int
for rem != 0 {
pg := f.m[pi]
if pg == nil {
pg = buffer.CGet(f.pgSize)
f.m[pi] = pg
}
nc = copy((*pg)[po:], b)
pi++
po = 0
rem -= nc
b = b[nc:]
}
f.size = mathutil.MaxInt64(f.size, off+int64(n))
return n, nil
}
type stat struct {
os.FileInfo
size int64
}
func (s stat) Size() int64 { return s.size }
type fileMap map[int64]mmap.MMap
type file struct {
f *os.File
m fileMap
maxPages int
pgBits uint
pgMask int
pgSize int
size int64
fsize int64
}
func newFile(f *os.File, maxSize int64, pgBits uint) (*file, error) {
if maxSize < 0 {
panic("internal error")
}
pgSize := 1 << pgBits
switch {
case sysPage > pgSize:
pgBits = uint(mathutil.Log2Uint64(uint64(sysPage)))
default:
pgBits = uint(mathutil.Log2Uint64(uint64(pgSize / sysPage * sysPage)))
}
pgSize = 1 << pgBits
fi := &file{
f: f,
m: fileMap{},
maxPages: int(mathutil.MinInt64(
1024,
mathutil.MaxInt64(maxSize/int64(pgSize), 1)),
),
pgBits: pgBits,
pgMask: pgSize - 1,
pgSize: pgSize,
}
info, err := f.Stat()
if err != nil {
return nil, err
}
if err = fi.Truncate(info.Size()); err != nil {
return nil, err
}
return fi, nil
}
func (f *file) ReadFrom(r io.Reader) (n int64, err error) { return readFrom(f, r) }
func (f *file) Sync() (err error) { return f.f.Sync() }
func (f *file) WriteTo(w io.Writer) (n int64, err error) { return writeTo(f, w) }
func (f *file) Close() (err error) {
for _, p := range f.m {
if err = p.Unmap(); err != nil {
return err
}
}
if err = f.f.Truncate(f.size); err != nil {
return err
}
if err = f.f.Sync(); err != nil {
return err
}
if err = f.f.Close(); err != nil {
return err
}
f.m = nil
f.f = nil
return nil
}
func (f *file) page(index int64) (mmap.MMap, error) {
if len(f.m) == f.maxPages {
for i, p := range f.m {
if err := p.Unmap(); err != nil {
return nil, err
}
delete(f.m, i)
break
}
}
off := index << f.pgBits
fsize := off + int64(f.pgSize)
if fsize > f.fsize {
if err := f.f.Truncate(fsize); err != nil {
return nil, err
}
f.fsize = fsize
}
p, err := mmap.MapRegion(f.f, f.pgSize, mmap.RDWR, 0, off)
if err != nil {
return nil, err
}
f.m[index] = p
return p, nil
}
func (f *file) ReadAt(b []byte, off int64) (n int, err error) {
avail := f.size - off
pi := off >> f.pgBits
po := int(off) & f.pgMask
rem := len(b)
if int64(rem) >= avail {
rem = int(avail)
err = io.EOF
}
for rem != 0 && avail > 0 {
pg := f.m[pi]
if pg == nil {
if pg, err = f.page(pi); err != nil {
return n, err
}
}
nc := copy(b[:mathutil.Min(rem, f.pgSize)], pg[po:])
pi++
po = 0
rem -= nc
n += nc
b = b[nc:]
}
return n, err
}
func (f *file) Stat() (os.FileInfo, error) {
fi, err := f.f.Stat()
if err != nil {
return nil, err
}
return stat{fi, f.size}, nil
}
func (f *file) Truncate(size int64) (err error) {
if size < 0 {
return fmt.Errorf("invalid truncate size: %d", size)
}
first := size >> f.pgBits
if size&int64(f.pgMask) != 0 {
first++
}
last := f.size >> f.pgBits
if f.size&int64(f.pgMask) != 0 {
last++
}
for ; first <= last; first++ {
if p := f.m[first]; p != nil {
if err := p.Unmap(); err != nil {
return err
}
}
delete(f.m, first)
}
f.size = size
fsize := (size + int64(f.pgSize) - 1) &^ int64(f.pgMask)
if fsize != f.fsize {
if err := f.f.Truncate(fsize); err != nil {
return err
}
}
f.fsize = fsize
return nil
}
func (f *file) WriteAt(b []byte, off int64) (n int, err error) {
pi := off >> f.pgBits
po := int(off) & f.pgMask
n = len(b)
rem := n
var nc int
for rem != 0 {
pg := f.m[pi]
if pg == nil {
pg, err = f.page(pi)
if err != nil {
return n, err
}
}
nc = copy(pg[po:], b)
pi++
po = 0
rem -= nc
b = b[nc:]
}
f.size = mathutil.MaxInt64(f.size, off+int64(n))
return n, nil
}
// ----------------------------------------------------------------------------
func readFrom(f Interface, r io.Reader) (n int64, err error) {
f.Truncate(0)
p := buffer.Get(copyBufSize)
b := *p
defer buffer.Put(p)
var off int64
var werr error
for {
rn, rerr := r.Read(b)
if rn != 0 {
_, werr = f.WriteAt(b[:rn], off)
n += int64(rn)
off += int64(rn)
}
if rerr != nil {
if !fileutil.IsEOF(rerr) {
err = rerr
}
break
}
if werr != nil {
err = werr
break
}
}
return n, err
}
func writeTo(f Interface, w io.Writer) (n int64, err error) {
p := buffer.Get(copyBufSize)
b := *p
defer buffer.Put(p)
var off int64
var werr error
for {
rn, rerr := f.ReadAt(b, off)
if rn != 0 {
_, werr = w.Write(b[:rn])
n += int64(rn)
off += int64(rn)
}
if rerr != nil {
if !fileutil.IsEOF(rerr) {
err = rerr
}
break
}
if werr != nil {
err = werr
break
}
}
return n, err
}

55
vendor/github.com/cznic/internal/slice/Makefile generated vendored
View File

@ -1,55 +0,0 @@
# Copyright 2016 The Internal Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.
.PHONY: all clean cover cpu editor internalError later mem nuke todo edit
grep=--include=*.go --include=*.l --include=*.y --include=*.yy
ngrep='TODOOK\|parser\.go\|scanner\.go\|.*_string\.go'
all: editor
go vet 2>&1 | grep -v $(ngrep) || true
golint 2>&1 | grep -v $(ngrep) || true
make todo
unused . || true
misspell *.go
gosimple || true
clean:
go clean
rm -f *~ *.test *.out
cover:
t=$(shell tempfile) ; go test -coverprofile $$t && go tool cover -html $$t && unlink $$t
cpu: clean
go test -run @ -bench . -cpuprofile cpu.out
go tool pprof -lines *.test cpu.out
edit:
@ 1>/dev/null 2>/dev/null gvim -p Makefile log *.go
editor:
gofmt -l -s -w *.go
go test 2>&1 | tee log
go build
internalError:
egrep -ho '"internal error.*"' *.go | sort | cat -n
later:
@grep -n $(grep) LATER * || true
@grep -n $(grep) MAYBE * || true
mem: clean
go test -run @ -bench . -memprofile mem.out -memprofilerate 1 -timeout 24h
go tool pprof -lines -web -alloc_space *.test mem.out
nuke: clean
go clean -i
todo:
@grep -nr $(grep) ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* * | grep -v $(ngrep) || true
@grep -nr $(grep) TODO * | grep -v $(ngrep) || true
@grep -nr $(grep) BUG * | grep -v $(ngrep) || true
@grep -nr $(grep) [^[:alpha:]]println * | grep -v $(ngrep) || true

173
vendor/github.com/cznic/internal/slice/pool.go generated vendored
View File

@ -1,173 +0,0 @@
// Copyright 2016 The Internal Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package slice implements pools of pointers to slices.
package slice
import (
"sync"
"github.com/cznic/mathutil"
)
var (
// Bytes is a ready to use *[]byte Pool.
Bytes *Pool
// Ints is a ready to use *[]int Pool.
Ints *Pool
)
func init() {
Bytes = newBytes()
Ints = NewPool(
func(size int) interface{} { // create
b := make([]int, size)
return &b
},
func(s interface{}) { // clear
b := *s.(*[]int)
b = b[:cap(b)]
for i := range b {
b[i] = 0
}
},
func(s interface{}, size int) { // setSize
p := s.(*[]int)
*p = (*p)[:size]
},
func(s interface{}) int { return cap(*s.(*[]int)) }, // cap
)
}
func newBytes() *Pool {
return NewPool(
func(size int) interface{} { // create
b := make([]byte, size)
return &b
},
func(s interface{}) { // clear
b := *s.(*[]byte)
b = b[:cap(b)]
for i := range b {
b[i] = 0
}
},
func(s interface{}, size int) { // setSize
p := s.(*[]byte)
*p = (*p)[:size]
},
func(s interface{}) int { return cap(*s.(*[]byte)) }, // cap
)
}
// Pool implements a pool of pointers to slices.
//
// Example usage pattern (assuming pool is, for example, a *[]byte Pool)
//
// p := pool.Get(size).(*[]byte)
// b := *p // Now you can use b in any way you need.
// ...
// // When b will not be used anymore
// pool.Put(p)
// ...
// // If b or p are not going out of scope soon, optionally
// b = nil
// p = nil
//
// Otherwise the pool cannot release the slice on garbage collection.
//
// Do not do
//
// p := pool.Get(size).(*[]byte)
// b := *p
// ...
// pool.Put(&b)
//
// or
//
// b := *pool.Get(size).(*[]byte)
// ...
// pool.Put(&b)
type Pool struct {
cap func(interface{}) int
clear func(interface{})
m [63]sync.Pool
null interface{}
setSize func(interface{}, int)
}
// NewPool returns a newly created Pool. Assuming the desired slice type is
// []T:
//
// The create function returns a *[]T of len == cap == size.
//
// The argument of clear is *[]T and the function sets all the slice elements
// to the respective zero value.
//
// The setSize function gets a *[]T and sets its len to size.
//
// The cap function gets a *[]T and returns its capacity.
func NewPool(
create func(size int) interface{},
clear func(interface{}),
setSize func(p interface{}, size int),
cap func(p interface{}) int,
) *Pool {
p := &Pool{clear: clear, setSize: setSize, cap: cap, null: create(0)}
for i := range p.m {
size := 1 << uint(i)
p.m[i] = sync.Pool{New: func() interface{} {
// 0: 1 - 1
// 1: 10 - 10
// 2: 11 - 100
// 3: 101 - 1000
// 4: 1001 - 10000
// 5: 10001 - 100000
return create(size)
}}
}
return p
}
// CGet returns a *[]T of len size. The pointed to slice is zeroed up to its
// cap. CGet panics for size < 0.
//
// CGet is safe for concurrent use by multiple goroutines.
func (p *Pool) CGet(size int) interface{} {
s := p.Get(size)
p.clear(s)
return s
}
// Get returns a *[]T of len size. The pointed to slice is not zeroed. Get
// panics for size < 0.
//
// Get is safe for concurrent use by multiple goroutines.
func (p *Pool) Get(size int) interface{} {
var index int
switch {
case size < 0:
panic("Pool.Get: negative size")
case size == 0:
return p.null
case size > 1:
index = mathutil.Log2Uint64(uint64(size-1)) + 1
}
s := p.m[index].Get()
p.setSize(s, size)
return s
}
// Put puts a *[]T into a pool for possible later reuse by CGet or Get. Put
// panics is its argument is not of type *[]T.
//
// Put is safe for concurrent use by multiple goroutines.
func (p *Pool) Put(b interface{}) {
size := p.cap(b)
if size == 0 {
return
}
p.m[mathutil.Log2Uint64(uint64(size))].Put(b)
}

11
vendor/github.com/cznic/kv/AUTHORS generated vendored
View File

@ -1,11 +0,0 @@
# This file lists authors for copyright purposes. This file is distinct from
# the CONTRIBUTORS files. See the latter for an explanation.
#
# Names should be added to this file as:
# Name or Organization <email address>
#
# The email address is not required for organizations.
#
# Please keep the list sorted.
Jan Mercl <0xjnml@gmail.com>

13
vendor/github.com/cznic/kv/CONTRIBUTORS generated vendored
View File

@ -1,13 +0,0 @@
# This file lists people who contributed code to this repository. The AUTHORS
# file lists the copyright holders; this file lists people.
#
# Names should be added to this file like so:
# Name <email address>
#
# Please keep the list sorted.
Brad Fitzpatrick <brad@danga.com>
Jan Mercl <0xjnml@gmail.com>
Patrick Mézard <patrick@mezard.eu>
Salmān Aljammāz <s@0x65.net>
Tamás Gulácsi <gt-dev@gthomas.eu>

27
vendor/github.com/cznic/kv/LICENSE generated vendored
View File

@ -1,27 +0,0 @@
Copyright (c) 2014 The kv Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the names of the authors nor the names of the
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

55
vendor/github.com/cznic/kv/Makefile generated vendored
View File

@ -1,55 +0,0 @@
# Copyright 2014 The kv Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.
.PHONY: all clean cover cpu editor internalError later mem nuke todo edit
grep=--include=*.go --include=*.l --include=*.y --include=*.yy
ngrep='TODOOK\|parser\.go\|scanner\.go\|.*_string\.go'
all: editor
go vet 2>&1 | grep -v $(ngrep) || true
golint 2>&1 | grep -v $(ngrep) || true
make todo
unused . || true
misspell *.go
gosimple || true
clean:
go clean
rm -f *~ *.test *.out _testdata/temp*
cover:
t=$(shell tempfile) ; go test -coverprofile $$t && go tool cover -html $$t && unlink $$t
cpu: clean
go test -run @ -bench . -cpuprofile cpu.out
go tool pprof -lines *.test cpu.out
edit:
gvim -p Makefile *.go
editor:
gofmt -l -s -w *.go
go test 2>&1 | tee log
go build
internalError:
egrep -ho '"internal error.*"' *.go | sort | cat -n
later:
@grep -n $(grep) LATER * || true
@grep -n $(grep) MAYBE * || true
mem: clean
go test -run @ -bench . -memprofile mem.out -memprofilerate 1 -timeout 24h
go tool pprof -lines -web -alloc_space *.test mem.out
nuke: clean
go clean -i
todo:
@grep -nr $(grep) ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* * || true
@grep -nr $(grep) TODO * || true
@grep -nr $(grep) BUG * || true
@grep -nr $(grep) [^[:alpha:]]println * || true

10
vendor/github.com/cznic/kv/README.md generated vendored
View File

@ -1,10 +0,0 @@
kv
==
Package kv implements a simple and easy to use persistent key/value (KV) store.
Installation
$ go get github.com/cznic/kv
Documentation: [godoc.org/github.com/cznic/kv](http://godoc.org/github.com/cznic/kv)

86
vendor/github.com/cznic/kv/doc.go generated vendored
View File

@ -1,86 +0,0 @@
// Copyright 2014 The kv Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
/*
Package kv implements a simple and easy to use persistent key/value (KV) store.
Changelog
2016-07-11: KV now uses the stable version of lldb. (github.com/cznic/lldb).
The stored KV pairs are sorted in the key collation order defined by an user
supplied 'compare' function (passed as a field in Options).
Keys and Values Limits
Keys, as well as the values associated with them, are opaque []bytes. Maximum
size of a "native" key or value is 65787 bytes. Larger keys or values have to
be composed of the "native" ones in client code.
Database limits
The maximum DB size kv can handle is 2^60 bytes (1 exabyte). See also [4]:
"Block handles".
ACID and transactional properties
Transactions are resource limited. All changes made by a transaction are held
in memory until the top level transaction is committed. ACID[1] implementation
notes/details follows.
Atomicity
A successfully committed transaction appears (by its effects on the database)
to be indivisible ("atomic") iff the transaction is performed in isolation. An
aborted (via RollBack) transaction appears like it never happened under the
same limitation.
Atomic updates to the DB, via functions like Set, Inc, etc., are performed in
their own automatic transaction. If the partial progress of any such function
fails at any point, the automatic transaction is canceled via Rollback before
returning from the function. A non nil error is returned in that case.
Consistency
All reads, including those made from any other concurrent non isolated
transaction(s), performed during a not yet committed transaction, are dirty
reads, i.e. the data returned are consistent with the in-progress state of the
open transaction, or all of the open transactions. Obviously, conflicts, data
races and inconsistent states can happen, but iff non isolated transactions are
performed.
Performing a Rollback at a nested transaction level properly returns the
transaction state (and data read from the DB) to what it was before the
respective BeginTransaction.
Isolation
Transactions of the atomic updating functions (Set, Put, Delete ...) are always
isolated. Transactions controlled by BeginTransaction/Commit/RollBack, are
isolated iff their execution is serialized.
Durability
Transactions are committed using the two phase commit protocol(2PC)[2] and a
write ahead log(WAL)[3]. DB recovery after a crash is performed automatically
using data from the WAL. Last transaction data, either of an in progress
transaction or a transaction being committed at the moment of the crash, can get
lost.
No protection from non readable files, files corrupted by other processes or by
memory faults or other HW problems, is provided. Always properly backup your DB
data file(s).
Links
Referenced from above:
[1]: http://en.wikipedia.org/wiki/ACID
[2]: http://en.wikipedia.org/wiki/2PC
[3]: http://en.wikipedia.org/wiki/Write_ahead_logging
[4]: http://godoc.org/github.com/cznic/lldb#Allocator
*/
package kv

31
vendor/github.com/cznic/kv/etc.go generated vendored
View File

@ -1,31 +0,0 @@
// Copyright 2014 The kv Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package kv
import (
"bytes"
"fmt"
)
type header struct {
magic []byte
ver byte
reserved []byte
}
func (h *header) rd(b []byte) error {
if len(b) != 16 {
panic("internal error")
}
if h.magic = b[:4]; !bytes.Equal(h.magic, []byte(magic)) {
return fmt.Errorf("Unknown file format")
}
b = b[4:]
h.ver = b[0]
h.reserved = b[1:]
return nil
}

851
vendor/github.com/cznic/kv/kv.go generated vendored
View File

@ -1,851 +0,0 @@
// Copyright 2014 The kv Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package kv
import (
"encoding/binary"
"fmt"
"io"
"os"
"sync"
"time"
"github.com/cznic/fileutil"
"github.com/cznic/internal/buffer"
"github.com/cznic/lldb"
)
const (
magic = "\x60\xdbKV"
)
const (
stDisabled = iota // stDisabled must be zero
stIdle
stCollecting
stIdleArmed
stCollectingArmed
stCollectingTriggered
stEndUpdateFailed
)
func init() {
if stDisabled != 0 {
panic("stDisabled != 0")
}
}
// DB represents the database (the KV store).
type DB struct {
acidNest int // Grace period nesting level
acidState int // Grace period FSM state.
acidTimer *time.Timer // Grace period timer
alloc *lldb.Allocator // The machinery. Wraps filer
bkl sync.Mutex // Big Kernel Lock
closeMu sync.Mutex // Close() coordination
closed bool // it was
filer lldb.Filer // Wraps f
gracePeriod time.Duration // WAL grace period
isMem bool // No signal capture
lastCommitErr error // from failed EndUpdate
lock io.Closer // The DB file lock
opts *Options
root *lldb.BTree // The KV layer
wal *os.File // WAL if any
}
// CreateFromFiler is like Create but accepts an arbitrary backing storage
// provided by filer.
//
// For the meaning of opts please see documentation of Options.
func CreateFromFiler(filer lldb.Filer, opts *Options) (db *DB, err error) {
opts = opts.clone()
opts._ACID = _ACIDFull
return create(filer, opts, false)
}
// Create creates the named DB file mode 0666 (before umask). The file must not
// already exist. If successful, methods on the returned DB can be used for
// I/O; the associated file descriptor has mode os.O_RDWR. If there is an
// error, it will be of type *os.PathError.
//
// For the meaning of opts please see documentation of Options.
func Create(name string, opts *Options) (db *DB, err error) {
opts = opts.clone()
opts._ACID = _ACIDFull
f, err := os.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0666)
if err != nil {
return
}
return CreateFromFiler(lldb.NewSimpleFileFiler(f), opts)
}
func create(filer lldb.Filer, opts *Options, isMem bool) (db *DB, err error) {
defer func() {
if db != nil {
db.opts = opts
}
}()
defer func() {
lock := opts.lock
if err != nil && lock != nil {
lock.Close()
db = nil
}
}()
if err = opts.check(filer.Name(), true, !isMem); err != nil {
return
}
b := [16]byte{byte(magic[0]), byte(magic[1]), byte(magic[2]), byte(magic[3]), 0x00} // ver 0x00
if n, err := filer.WriteAt(b[:], 0); n != 16 {
return nil, &os.PathError{Op: "kv.create.WriteAt", Path: filer.Name(), Err: err}
}
db = &DB{lock: opts.lock}
filer = lldb.NewInnerFiler(filer, 16)
if filer, err = opts.acidFiler(db, filer); err != nil {
return nil, err
}
db.filer = filer
if err = filer.BeginUpdate(); err != nil {
return
}
defer func() {
if e := filer.EndUpdate(); e != nil {
if err == nil {
err = e
}
}
}()
if db.alloc, err = lldb.NewAllocator(filer, &lldb.Options{}); err != nil {
return nil, &os.PathError{Op: "kv.create", Path: filer.Name(), Err: err}
}
db.alloc.Compress = true
db.isMem = isMem
var h int64
if db.root, h, err = lldb.CreateBTree(db.alloc, opts.Compare); err != nil {
return
}
if h != 1 {
panic("internal error")
}
db.wal = opts.wal
return
}
// CreateMem creates a new instance of an in-memory DB not backed by a disk
// file. Memory DBs are resource limited as they are completely held in memory
// and are not automatically persisted.
//
// For the meaning of opts please see documentation of Options.
func CreateMem(opts *Options) (db *DB, err error) {
opts = opts.clone()
opts._ACID = _ACIDTransactions
f := lldb.NewMemFiler()
return create(f, opts, true)
}
// CreateTemp creates a new temporary DB in the directory dir with a basename
// beginning with prefix and name ending in suffix. If dir is the empty string,
// CreateTemp uses the default directory for temporary files (see os.TempDir).
// Multiple programs calling CreateTemp simultaneously will not choose the same
// file name for the DB. The caller can use Name() to find the pathname of the
// DB file. It is the caller's responsibility to remove the file when no longer
// needed.
//
// For the meaning of opts please see documentation of Options.
func CreateTemp(dir, prefix, suffix string, opts *Options) (db *DB, err error) {
opts = opts.clone()
opts._ACID = _ACIDFull
f, err := fileutil.TempFile(dir, prefix, suffix)
if err != nil {
return
}
return create(lldb.NewSimpleFileFiler(f), opts, false)
}
// Open opens the named DB file for reading/writing. If successful, methods on
// the returned DB can be used for I/O; the associated file descriptor has mode
// os.O_RDWR. If there is an error, it will be of type *os.PathError.
//
// Note: While a DB is opened, it is locked and cannot be simultaneously opened
// again.
//
// For the meaning of opts please see documentation of Options.
func Open(name string, opts *Options) (db *DB, err error) {
f, err := os.OpenFile(name, os.O_RDWR, 0666)
if err != nil {
return nil, err
}
return OpenFromFiler(lldb.NewSimpleFileFiler(f), opts)
}
// OpenFromFiler is like Open but it accepts an arbitrary backing storage
// provided by filer.
func OpenFromFiler(filer lldb.Filer, opts *Options) (db *DB, err error) {
opts = opts.clone()
opts._ACID = _ACIDFull
defer func() {
if db != nil {
db.opts = opts
}
}()
defer func() {
lock := opts.lock
if err != nil && lock != nil {
lock.Close()
db = nil
}
if err != nil {
if db != nil {
db.Close()
db = nil
}
}
}()
name := filer.Name()
if err = opts.check(name, false, true); err != nil {
return
}
sz, err := filer.Size()
if err != nil {
return
}
if sz%16 != 0 {
return nil, &os.PathError{Op: "kv.Open:", Path: name, Err: fmt.Errorf("file size %d(%#x) is not 0 (mod 16)", sz, sz)}
}
var b [16]byte
if n, err := filer.ReadAt(b[:], 0); n != 16 || err != nil {
return nil, &os.PathError{Op: "kv.Open.ReadAt", Path: name, Err: err}
}
var h header
if err = h.rd(b[:]); err != nil {
return nil, &os.PathError{Op: "kv.Open:validate header", Path: name, Err: err}
}
db = &DB{lock: opts.lock}
if filer, err = opts.acidFiler(db, filer); err != nil {
return nil, err
}
db.filer = filer
switch h.ver {
default:
return nil, &os.PathError{Op: "kv.Open", Path: name, Err: fmt.Errorf("unknown/unsupported kv file format version %#x", h.ver)}
case 0x00:
if _, err = open00(name, db); err != nil {
return nil, err
}
}
db.root, err = lldb.OpenBTree(db.alloc, opts.Compare, 1)
db.wal = opts.wal
if opts.VerifyDbAfterOpen {
err = verifyAllocator(db.alloc)
}
return
}
// Close closes the DB, rendering it unusable for I/O. It returns an error, if
// any. Failing to call Close before exiting a program can lose the last open
// or being committed transaction.
//
// Successful Close is idempotent.
func (db *DB) Close() (err error) {
db.closeMu.Lock()
defer db.closeMu.Unlock()
if db.closed {
return
}
db.closed = true
if err = db.enter(); err != nil {
return
}
doLeave := true
defer func() {
db.wal = nil
if e := recover(); e != nil {
err = fmt.Errorf("%v", e)
}
if doLeave {
db.leave(&err)
}
}()
if db.acidTimer != nil {
db.acidTimer.Stop()
}
var e error
for db.acidNest > 0 {
db.acidNest--
if e = db.filer.EndUpdate(); err == nil {
err = e
}
}
doLeave = false
if e = db.leave(&err); err == nil {
err = e
}
if db.opts.VerifyDbBeforeClose {
if e = verifyAllocator(db.alloc); err == nil {
err = e
}
}
if e = db.close(); err == nil {
err = e
}
if lock := db.lock; lock != nil {
if e = lock.Close(); err == nil {
err = e
}
}
if wal := db.wal; wal != nil {
e = wal.Close()
db.wal = nil
if err == nil {
err = e
}
}
return
}
func (db *DB) close() (err error) {
// We are safe to close due to locked db.closeMu, but not safe against
// any other goroutine concurrently calling other exported db methods,
// causing a race[0] in the db.enter() mechanism. So we must lock
// db.bkl.
//
// [0]: https://github.com/cznic/kv/issues/17#issuecomment-31960658
db.bkl.Lock()
defer db.bkl.Unlock()
if db.isMem { // lldb.MemFiler
return
}
err = db.filer.Sync()
if e := db.filer.Close(); err == nil {
err = e
}
if db.opts.VerifyDbAfterClose {
if e := verifyDbFile(db.Name()); err == nil {
err = e
}
}
return
}
// Name returns the name of the DB file.
func (db *DB) Name() string {
return db.filer.Name()
}
// Size returns the size of the DB file.
func (db *DB) Size() (sz int64, err error) {
db.bkl.Lock()
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("%v", e)
}
db.bkl.Unlock()
}()
return db.filer.Size()
}
func (db *DB) enter() (err error) {
db.bkl.Lock()
switch db.acidState {
default:
panic("internal error")
case stDisabled:
db.acidNest++
if db.acidNest == 1 {
if err = db.filer.BeginUpdate(); err != nil {
return err
}
}
case stIdle:
if err = db.filer.BeginUpdate(); err != nil {
return err
}
db.acidNest = 1
db.acidTimer = time.AfterFunc(db.gracePeriod, db.timeout)
db.acidState = stCollecting
case stCollecting:
db.acidNest++
case stIdleArmed:
db.acidNest = 1
db.acidState = stCollectingArmed
case stCollectingArmed:
db.acidNest++
case stCollectingTriggered:
db.acidNest++
case stEndUpdateFailed:
return db.leave(&err)
}
return nil
}
func (db *DB) leave(err *error) error {
switch db.acidState {
default:
panic("internal error")
case stDisabled:
db.acidNest--
if db.acidNest == 0 {
if e := db.filer.EndUpdate(); e != nil && *err == nil {
*err = e
}
}
case stCollecting:
db.acidNest--
if db.acidNest == 0 {
db.acidState = stIdleArmed
}
case stCollectingArmed:
db.acidNest--
if db.acidNest == 0 {
db.acidState = stIdleArmed
}
case stCollectingTriggered:
db.acidNest--
if db.acidNest == 0 {
if e := db.filer.EndUpdate(); e != nil && *err == nil {
*err = e
}
db.acidState = stIdle
}
case stEndUpdateFailed:
db.bkl.Unlock()
return fmt.Errorf("Last transaction commit failed: %v", db.lastCommitErr)
}
if *err != nil {
db.filer.Rollback() // return the original, input error
}
db.bkl.Unlock()
return *err
}
func (db *DB) timeout() {
db.closeMu.Lock()
defer db.closeMu.Unlock()
if db.closed {
return
}
db.bkl.Lock()
defer db.bkl.Unlock()
switch db.acidState {
default:
panic("internal error")
case stIdle:
panic("internal error")
case stCollecting:
db.acidState = stCollectingTriggered
case stIdleArmed:
if err := db.filer.EndUpdate(); err != nil { // If EndUpdate fails, no WAL was written (automatic Rollback)
db.acidState = stEndUpdateFailed
db.lastCommitErr = err
return
}
db.acidState = stIdle
case stCollectingArmed:
db.acidState = stCollectingTriggered
case stCollectingTriggered:
panic("internal error")
}
}
// BeginTransaction starts a new transaction. Every call to BeginTransaction
// must be eventually "balanced" by exactly one call to Commit or Rollback (but
// not both). Calls to BeginTransaction may nest.
//
// BeginTransaction is atomic and it is safe for concurrent use by multiple
// goroutines (if/when that makes sense).
func (db *DB) BeginTransaction() (err error) {
if err = db.enter(); err != nil {
return
}
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("%v", e)
}
db.leave(&err)
}()
db.acidNest++
return db.filer.BeginUpdate()
}
// Commit commits the current transaction. If the transaction is the top level
// one, then all of the changes made within the transaction are atomically made
// persistent in the DB. Invocation of an unbalanced Commit is an error.
//
// Commit is atomic and it is safe for concurrent use by multiple goroutines
// (if/when that makes sense).
func (db *DB) Commit() (err error) {
if err = db.enter(); err != nil {
return
}
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("%v", e)
}
db.leave(&err)
}()
db.acidNest--
return db.filer.EndUpdate()
}
// Rollback cancels and undoes the innermost transaction level. If the
// transaction is the top level one, then no of the changes made within the
// transactions are persisted. Invocation of an unbalanced Rollback is an
// error.
//
// Rollback is atomic and it is safe for concurrent use by multiple goroutines
// (if/when that makes sense).
func (db *DB) Rollback() (err error) {
if err = db.enter(); err != nil {
return
}
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("%v", e)
}
db.leave(&err)
}()
db.acidNest--
return db.filer.Rollback()
}
// Verify attempts to find any structural errors in DB wrt the organization of
// it as defined by lldb.Allocator. Any problems found are reported to 'log'
// except non verify related errors like disk read fails etc. If 'log' returns
// false or the error doesn't allow to (reliably) continue, the verification
// process is stopped and an error is returned from the Verify function.
// Passing a nil log works like providing a log function always returning
// false. Any non-structural errors, like for instance Filer read errors, are
// NOT reported to 'log', but returned as the Verify's return value, because
// Verify cannot proceed in such cases. Verify returns nil only if it fully
// completed verifying DB without detecting any error.
//
// It is recommended to limit the number reported problems by returning false
// from 'log' after reaching some limit. Huge and corrupted DB can produce an
// overwhelming error report dataset.
//
// The verifying process will scan the whole DB at least 3 times (a trade
// between processing space and time consumed). It doesn't read the content of
// free blocks above the head/tail info bytes. If the 3rd phase detects lost
// free space, then a 4th scan (a faster one) is performed to precisely report
// all of them.
//
// Statistics are returned via 'stats' if non nil. The statistics are valid
// only if Verify succeeded, ie. it didn't reported anything to log and it
// returned a nil error.
func (db *DB) Verify(log func(error) bool, stats *lldb.AllocStats) (err error) {
bitmapf, err := fileutil.TempFile("", "verifier", ".tmp")
if err != nil {
return
}
defer func() {
tn := bitmapf.Name()
bitmapf.Close()
os.Remove(tn)
}()
bitmap := lldb.NewSimpleFileFiler(bitmapf)
if err = db.enter(); err != nil {
return
}
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("%v", e)
}
db.leave(&err)
}()
return db.alloc.Verify(bitmap, log, stats)
}
// Delete deletes key and its associated value from the DB.
//
// Delete is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) Delete(key []byte) (err error) {
if err = db.enter(); err != nil {
return
}
err = db.root.Delete(key)
return db.leave(&err)
}
// Extract is a combination of Get and Delete. If the key exists in the DB, it
// is returned (like Get) and also deleted from the DB in a more efficient way
// which doesn't search for the key twice. The returned slice may be a
// sub-slice of buf if buf was large enough to hold the entire content.
// Otherwise, a newly allocated slice will be returned. It is valid to pass a
// nil buf.
//
// Extract is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) Extract(buf, key []byte) (value []byte, err error) {
if err = db.enter(); err != nil {
return
}
value, err = db.root.Extract(buf, key)
db.leave(&err)
return
}
// First returns the first KV pair in the DB, if it exists. Otherwise key ==
// nil and value == nil.
//
// First is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) First() (key, value []byte, err error) {
db.bkl.Lock()
defer db.bkl.Unlock()
return db.root.First()
}
// Get returns the value associated with key, or nil if no such value exists.
// The returned slice may be a sub-slice of buf if buf was large enough to hold
// the entire content. Otherwise, a newly allocated slice will be returned. It
// is valid to pass a nil buf.
//
// Get is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) Get(buf, key []byte) (value []byte, err error) {
db.bkl.Lock()
defer db.bkl.Unlock()
return db.root.Get(buf, key)
}
// Last returns the last KV pair of the DB, if it exists. Otherwise key ==
// nil and value == nil.
//
// Last is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) Last() (key, value []byte, err error) {
db.bkl.Lock()
defer db.bkl.Unlock()
return db.root.Last()
}
// Put combines Get and Set in a more efficient way where the DB is searched
// for the key only once. The upd(ater) receives the current (key, old-value),
// if that exists or (key, nil) otherwise. It can then return a (new-value,
// true, nil) to create or overwrite the existing value in the KV pair, or
// (whatever, false, nil) if it decides not to create or not to update the
// value of the KV pair.
//
// db.Set(k, v)
//
// conceptually equals
//
// db.Put(k, func(k, v []byte){ return v, true }([]byte, bool))
//
// modulo the differing return values.
//
// The returned slice may be a sub-slice of buf if buf was large enough to hold
// the entire content. Otherwise, a newly allocated slice will be returned. It
// is valid to pass a nil buf.
//
// Put is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) Put(buf, key []byte, upd func(key, old []byte) (new []byte, write bool, err error)) (old []byte, written bool, err error) {
if err = db.enter(); err != nil {
return
}
old, written, err = db.root.Put(buf, key, upd)
db.leave(&err)
return
}
// Seek returns an enumerator positioned on the first key/value pair whose key
// is 'greater than or equal to' the given key. There may be no such pair, in
// which case the Next,Prev methods of the returned enumerator will always
// return io.EOF.
//
// Seek is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) Seek(key []byte) (enum *Enumerator, hit bool, err error) {
db.bkl.Lock()
defer db.bkl.Unlock()
enum0, hit, err := db.root.Seek(key)
if err != nil {
return
}
enum = &Enumerator{
db: db,
enum: enum0,
}
return
}
// SeekFirst returns an enumerator positioned on the first KV pair in the DB,
// if any. For an empty DB, err == io.EOF is returned.
//
// SeekFirst is atomic and it is safe for concurrent use by multiple
// goroutines.
func (db *DB) SeekFirst() (enum *Enumerator, err error) {
db.bkl.Lock()
defer db.bkl.Unlock()
enum0, err := db.root.SeekFirst()
if err != nil {
return
}
enum = &Enumerator{
db: db,
enum: enum0,
}
return
}
// SeekLast returns an enumerator positioned on the last KV pair in the DB,
// if any. For an empty DB, err == io.EOF is returned.
//
// SeekLast is atomic and it is safe for concurrent use by multiple
// goroutines.
func (db *DB) SeekLast() (enum *Enumerator, err error) {
db.bkl.Lock()
defer db.bkl.Unlock()
enum0, err := db.root.SeekLast()
if err != nil {
return
}
enum = &Enumerator{
db: db,
enum: enum0,
}
return
}
// Set sets the value associated with key. Any previous value, if existed, is
// overwritten by the new one.
//
// Set is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) Set(key, value []byte) (err error) {
if err = db.enter(); err != nil {
return
}
err = db.root.Set(key, value)
db.leave(&err)
return
}
// Enumerator captures the state of enumerating a DB. It is returned from the
// Seek* methods. Multiple enumerations may be in progress simultaneously. The
// enumerator is aware of any mutations made to the tree in the process of
// enumerating it and automatically resumes the enumeration.
//
// Multiple concurrently executing enumerations may be in progress.
type Enumerator struct {
db *DB
enum *lldb.BTreeEnumerator
}
// Next returns the currently enumerated KV pair, if it exists and moves to the
// next KV in the key collation order. If there is no KV pair to return, err ==
// io.EOF is returned.
//
// Next is atomic and it is safe for concurrent use by multiple goroutines.
func (e *Enumerator) Next() (key, value []byte, err error) {
e.db.bkl.Lock()
defer e.db.bkl.Unlock()
return e.enum.Next()
}
// Prev returns the currently enumerated KV pair, if it exists and moves to the
// previous KV in the key collation order. If there is no KV pair to return,
// err == io.EOF is returned.
//
// Prev is atomic and it is safe for concurrent use by multiple goroutines.
func (e *Enumerator) Prev() (key, value []byte, err error) {
e.db.bkl.Lock()
defer e.db.bkl.Unlock()
return e.enum.Prev()
}
// Inc atomically increments the value associated with key by delta and
// returns the new value. If the value doesn't exists before calling Inc or if
// the value is not an [8]byte, the value is considered to be zero before peforming Inc.
//
// Inc is atomic and it is safe for concurrent use by multiple goroutines.
func (db *DB) Inc(key []byte, delta int64) (val int64, err error) {
if err = db.enter(); err != nil {
return
}
defer db.leave(&err)
pbuf := buffer.Get(8)
defer buffer.Put(pbuf)
_, _, err = db.root.Put(
*pbuf,
key,
func(key []byte, old []byte) (new []byte, write bool, err error) {
write = true
if len(old) == 8 {
val = int64(binary.BigEndian.Uint64(old))
} else {
old = make([]byte, 8)
val = 0
}
val += delta
binary.BigEndian.PutUint64(old, uint64(val))
new = old
return
},
)
return
}
// WALName returns the name of the WAL file in use or an empty string for memory
// or closed databases.
func (db *DB) WALName() string {
if f := db.wal; f != nil {
return f.Name()
}
return ""
}

58
vendor/github.com/cznic/kv/lock.go generated vendored
View File

@ -1,58 +0,0 @@
// Copyright 2014 The kv Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package kv
import (
"crypto/sha1"
"fmt"
"io"
"os"
"path/filepath"
"sync"
)
func lockName(dbname string) string {
base := filepath.Base(filepath.Clean(dbname)) + "lockfile"
h := sha1.New()
io.WriteString(h, base)
return filepath.Join(filepath.Dir(dbname), fmt.Sprintf(".%x", h.Sum(nil)))
}
func defaultLocker(dbname string) (io.Closer, error) {
lname := lockName(dbname)
abs, err := filepath.Abs(lname)
if err != nil {
return nil, err
}
f, err := os.OpenFile(abs, os.O_CREATE|os.O_EXCL|os.O_RDONLY, 0666)
if os.IsExist(err) {
return nil, fmt.Errorf("cannot access DB %q: lock file %q exists", dbname, abs)
}
if err != nil {
return nil, err
}
return &lockCloser{f: f, abs: abs}, nil
}
type lockCloser struct {
f *os.File
abs string
once sync.Once
err error
}
func (lc *lockCloser) Close() error {
lc.once.Do(lc.close)
return lc.err
}
func (lc *lockCloser) close() {
if err := lc.f.Close(); err != nil {
lc.err = err
}
if err := os.Remove(lc.abs); err != nil {
lc.err = err
}
}

246
vendor/github.com/cznic/kv/options.go generated vendored
View File

@ -1,246 +0,0 @@
// Copyright 2014 The kv Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package kv
import (
"crypto/sha1"
"fmt"
"io"
"os"
"path/filepath"
"time"
"github.com/cznic/lldb"
)
const (
// BeginUpdate/EndUpdate/Rollback will be no-ops. All operations
// updating a DB will be written immediately including partial updates
// during operation's progress. If any update fails, the DB can become
// unusable. The same applies to DB crashes and/or any other non clean
// DB shutdown.
_ACIDNone = iota
// Enable transactions. BeginUpdate/EndUpdate/Rollback will be
// effective. All operations on the DB will be automatically performed
// within a transaction. Operations will thus either succeed completely
// or have no effect at all - they will be rollbacked in case of any
// error. If any update fails the DB will not be corrupted. DB crashes
// and/or any other non clean DB shutdown may still render the DB
// unusable.
_ACIDTransactions
// Enable durability. Same as ACIDTransactions plus enables 2PC and
// WAL. Updates to the DB will be first made permanent in a WAL and
// only after that reflected in the DB. A DB will automatically recover
// from crashes and/or any other non clean DB shutdown. Only last
// uncommitted transaction (transaction in progress ATM of a crash) can
// get lost.
//
// NOTE: Options.GracePeriod may extend the span of a single
// transaction to a batch of multiple transactions.
//
// NOTE2: Non zero GracePeriod requires GOMAXPROCS > 1 to work. Dbm
// checks GOMAXPROCS in such case and if the value is 1 it
// automatically sets GOMAXPROCS = 2.
_ACIDFull
)
// Options are passed to the DB create/open functions to amend the behavior of
// those functions. The compatibility promise is the same as of struct types in
// the Go standard library - introducing changes can be made only by adding new
// exported fields, which is backward compatible as long as client code uses
// field names to assign values of imported struct types literals.
type Options struct {
// Compare compares x and y. Compare may be nil, then bytes.Compare is
// used instead.
//
// Compare returns:
//
// -1 if x < y
// 0 if x == y
// +1 if x > y
Compare func(x, y []byte) int
// Locker specifies a function to lock a named file.
// On success it returns an io.Closer to release the lock.
// If nil, a default implementation is used.
Locker func(name string) (io.Closer, error)
// See the ACID* constants documentation.
_ACID int
// The write ahead log pathname. Applicable iff ACID == ACIDFull. May
// be left empty in which case an unspecified pathname will be chosen,
// which is computed from the DB name and which will be in the same
// directory as the DB. Moving or renaming the DB while it is shut down
// will break it's connection to the automatically computed name.
// Moving both the files (the DB and the WAL) into another directory
// with no renaming is safe.
//
// On creating a new DB the WAL file must not exist or it must be
// empty. It's not safe to write to a non empty WAL file as it may
// contain unprocessed DB recovery data.
WAL string
// Time to collect transactions before committing them into the WAL.
// Applicable iff ACID == ACIDFull. All updates are held in memory
// during the grace period so it should not be more than few seconds at
// most.
//
// Recommended value for GracePeriod is 1 second.
//
// NOTE: Using small GracePeriod values will make DB updates very slow.
// Zero GracePeriod will make every single update a separate 2PC/WAL
// transaction. Values smaller than about 100-200 milliseconds
// (particularly for mechanical, rotational HDs) are not recommended
// and they may not be always honored.
_GracePeriod time.Duration
wal *os.File
lock io.Closer
noClone bool // test hook
// VerifyDbBeforeOpen turns on structural verification of the DB before
// it is opened. This verification may legitimately fail if the DB
// crashed and a yet-to-be-processed non empty WAL file exists.
VerifyDbBeforeOpen bool
// VerifyDbAfterOpen turns on structural verification of the DB after
// it is opened and possibly recovered from WAL.
VerifyDbAfterOpen bool
// VerifyDbBeforeClose turns on structural verification of the DB
// before it is closed.
VerifyDbBeforeClose bool
// VerifyDbAfterClose turns on structural verification of the DB after
// it is closed.
VerifyDbAfterClose bool
// Turns on verification of every single mutation of the DB. Before any
// such mutation a snapshot of the DB is created and the specific
// mutation operation and parameters are recorded. After the mutation
// the whole DB is verified. If the verification fails the last known
// good state (the snapshot discussed above) and the corrupted state
// are "core" dumped to a well known location (TBD).
//
//MAYBE ParanoidUpdates bool
}
func (o *Options) locker(dbname string) (io.Closer, error) {
if o == nil || o.Locker == nil {
return defaultLocker(dbname)
}
return o.Locker(dbname)
}
func (o *Options) clone() *Options {
if o.noClone {
return o
}
r := &Options{}
*r = *o
return r
}
func (o *Options) check(dbname string, new, lock bool) (err error) {
if lock {
if o.lock, err = o.locker(dbname); err != nil {
return
}
}
if o.VerifyDbBeforeOpen && !new {
if err = verifyDbFile(dbname); err != nil {
return
}
}
switch o._ACID {
default:
panic("internal error")
case _ACIDTransactions:
case _ACIDFull:
o._GracePeriod = time.Second
if o.WAL == "" {
o.WAL = o.walName(dbname, o.WAL)
}
switch new {
case true:
if o.wal, err = os.OpenFile(o.WAL, os.O_CREATE|os.O_EXCL|os.O_RDWR, 0666); err != nil {
if os.IsExist(err) {
fi, e := os.Stat(o.WAL)
if e != nil {
return e
}
if sz := fi.Size(); sz != 0 {
return fmt.Errorf("cannot create DB %q: non empty WAL file %q (size %d) exists", dbname, o.WAL, sz)
}
o.wal, err = os.OpenFile(o.WAL, os.O_RDWR, 0666)
}
return
}
case false:
if o.wal, err = os.OpenFile(o.WAL, os.O_RDWR, 0666); err != nil {
if os.IsNotExist(err) {
if o.wal, err = os.OpenFile(o.WAL, os.O_CREATE|os.O_EXCL|os.O_RDWR, 0666); err != nil {
return fmt.Errorf("cannot open DB %q: failed to create WAL file %q: %v", dbname, o.WAL, err)
}
err = nil
}
return err
}
}
}
return err
}
func (o *Options) walName(dbname, wal string) (r string) {
if wal != "" {
return filepath.Clean(wal)
}
base := filepath.Base(filepath.Clean(dbname))
h := sha1.New()
io.WriteString(h, base)
return filepath.Join(filepath.Dir(dbname), fmt.Sprintf(".%x", h.Sum(nil)))
}
func (o *Options) acidFiler(db *DB, f lldb.Filer) (r lldb.Filer, err error) {
switch o._ACID {
default:
panic("internal error")
case _ACIDTransactions:
if r, err = lldb.NewRollbackFiler(
f,
func(sz int64) error {
return f.Truncate(sz)
},
f,
); err != nil {
return nil, err
}
return r, nil
case _ACIDFull:
if r, err = lldb.NewACIDFiler(f, o.wal); err != nil {
return nil, err
}
db.acidState = stIdle
db.gracePeriod = o._GracePeriod
if o._GracePeriod == 0 {
panic("internal error")
}
return r, nil
}
}

35
vendor/github.com/cznic/kv/perf-4670.log generated vendored
View File

@ -1,35 +0,0 @@
$ benchcmp -mag -changed log-bench-2016-02-08-2109-120f703e log-bench-2016-07-19-1813-32e56c29-lldb-2016-07-24-1458-74c3b196
benchmark old ns/op new ns/op delta
BenchmarkFirst16-4 4715 3680 -21.95%
BenchmarkSet16-4 105255 88470 -15.95%
BenchmarkPut16-4 121708 104760 -13.93%
BenchmarkGet16-4 108211 93534 -13.56%
BenchmarkDelete16-4 111052 96004 -13.55%
BenchmarkExtract16-4 107820 93991 -12.83%
BenchmarkSeek-4 104365 93677 -10.24%
BenchmarkLast16-4 4075 3671 -9.91%
BenchmarkInc-4 2207 2046 -7.29%
BenchmarkEnumerateDB-4 231182 227247 -1.70%
BenchmarkNext1e3-4 222782 222161 -0.28%
benchmark old allocs new allocs delta
BenchmarkFirst16-4 4 6 +50.00%
BenchmarkLast16-4 4 6 +50.00%
BenchmarkSeek-4 19 22 +15.79%
BenchmarkPut16-4 15 14 -6.67%
BenchmarkDelete16-4 18 19 +5.56%
BenchmarkGet16-4 18 19 +5.56%
BenchmarkEnumerateDB-4 3008 3010 +0.07%
benchmark old bytes new bytes delta
BenchmarkDelete16-4 10193 502 -95.08%
BenchmarkGet16-4 147867 31569 -78.65%
BenchmarkSeek-4 133654 31436 -76.48%
BenchmarkPut16-4 1127 360 -68.06%
BenchmarkExtract16-4 1042 360 -65.45%
BenchmarkSet16-4 874 350 -59.95%
BenchmarkFirst16-4 24944 16730 -32.93%
BenchmarkLast16-4 20848 16730 -19.75%
BenchmarkEnumerateDB-4 65159 57065 -12.42%
BenchmarkInc-4 32 36 +12.50%
BenchmarkNext1e3-4 24002 24000 -0.01%

35
vendor/github.com/cznic/kv/perf-r550.log generated vendored
View File

@ -1,35 +0,0 @@
$ benchcmp -mag -changed log-bench-2016-02-08-2109-120f703e log-bench-2016-07-19-1813-32e56c29-lldb-2016-07-24-1458-74c3b196
benchmark old ns/op new ns/op delta
BenchmarkGet16-4 263400 180228 -31.58%
BenchmarkSeek-4 256307 181534 -29.17%
BenchmarkDelete16-4 237945 181734 -23.62%
BenchmarkFirst16-4 16199 12430 -23.27%
BenchmarkExtract16-4 231676 189059 -18.40%
BenchmarkLast16-4 13875 11783 -15.08%
BenchmarkPut16-4 206700 177933 -13.92%
BenchmarkSet16-4 174694 154715 -11.44%
BenchmarkInc-4 5907 5565 -5.79%
BenchmarkEnumerateDB-4 432438 427689 -1.10%
BenchmarkNext1e3-4 421038 421171 +0.03%
benchmark old allocs new allocs delta
BenchmarkFirst16-4 4 6 +50.00%
BenchmarkLast16-4 4 6 +50.00%
BenchmarkPut16-4 15 13 -13.33%
BenchmarkSeek-4 19 21 +10.53%
BenchmarkSet16-4 14 13 -7.14%
BenchmarkExtract16-4 17 16 -5.88%
BenchmarkEnumerateDB-4 3007 3010 +0.10%
benchmark old bytes new bytes delta
BenchmarkDelete16-4 10118 391 -96.14%
BenchmarkGet16-4 147861 30234 -79.55%
BenchmarkPut16-4 1631 334 -79.52%
BenchmarkExtract16-4 1329 300 -77.43%
BenchmarkSeek-4 133640 30385 -77.26%
BenchmarkSet16-4 1293 399 -69.14%
BenchmarkFirst16-4 24945 16730 -32.93%
BenchmarkLast16-4 20849 16730 -19.76%
BenchmarkEnumerateDB-4 65121 57066 -12.37%
BenchmarkInc-4 32 36 +12.50%
BenchmarkNext1e3-4 24008 24006 -0.01%

21
vendor/github.com/cznic/kv/v0.go generated vendored
View File

@ -1,21 +0,0 @@
// Copyright 2014 The kv Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package kv
import (
"os"
"github.com/cznic/lldb"
)
func open00(name string, in *DB) (db *DB, err error) {
db = in
if db.alloc, err = lldb.NewAllocator(lldb.NewInnerFiler(db.filer, 16), &lldb.Options{}); err != nil {
return nil, &os.PathError{Op: "kv.open00", Path: name, Err: err}
}
db.alloc.Compress = true
return
}

89
vendor/github.com/cznic/kv/verify.go generated vendored
View File

@ -1,89 +0,0 @@
// Copyright 2014 The kv Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package kv
import (
"fmt"
"io"
"io/ioutil"
"os"
"github.com/cznic/lldb"
)
func verifyAllocator(a *lldb.Allocator) error {
bits, err := ioutil.TempFile("", "kv-verify-")
if err != nil {
return err
}
sf := lldb.NewSimpleFileFiler(bits)
defer func() {
nm := bits.Name()
sf.Close()
os.Remove(nm)
}()
var lerr error
if err = a.Verify(
sf,
func(err error) bool {
lerr = err
return false
},
nil,
); err != nil {
return err
}
if lerr != nil {
return lerr
}
t, err := lldb.OpenBTree(a, nil, 1)
if err != nil {
return err
}
e, err := t.SeekFirst()
if err != nil {
if err == io.EOF {
err = nil
}
return err
}
for {
_, _, err := e.Next()
if err != nil {
if err == io.EOF {
err = nil
}
return err
}
}
}
func verifyDbFile(fn string) error {
f, err := os.OpenFile(fn, os.O_RDWR, 0666)
if err != nil {
return err
}
sf := lldb.NewSimpleFileFiler(f)
if f == nil {
return fmt.Errorf("cannot create %s", fn)
}
defer sf.Close()
a, err := lldb.NewAllocator(lldb.NewInnerFiler(sf, 16), &lldb.Options{})
if err != nil {
return err
}
return verifyAllocator(a)
}

400
vendor/github.com/cznic/lldb/2pc.go generated vendored
View File

@ -1,400 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Two Phase Commit & Structural ACID
package lldb
import (
"bufio"
"encoding/binary"
"fmt"
"io"
"os"
"github.com/cznic/fileutil"
"github.com/cznic/mathutil"
)
var _ Filer = &ACIDFiler0{} // Ensure ACIDFiler0 is a Filer
type acidWrite struct {
b []byte
off int64
}
type acidWriter0 ACIDFiler0
func (a *acidWriter0) WriteAt(b []byte, off int64) (n int, err error) {
f := (*ACIDFiler0)(a)
if f.newEpoch {
f.newEpoch = false
f.data = f.data[:0]
if err = a.writePacket([]interface{}{wpt00Header, walTypeACIDFiler0, ""}); err != nil {
return
}
}
if err = a.writePacket([]interface{}{wpt00WriteData, b, off}); err != nil {
return
}
f.data = append(f.data, acidWrite{b, off})
return len(b), nil
}
func (a *acidWriter0) writePacket(items []interface{}) (err error) {
f := (*ACIDFiler0)(a)
b, err := EncodeScalars(items...)
if err != nil {
return
}
var b4 [4]byte
binary.BigEndian.PutUint32(b4[:], uint32(len(b)))
if _, err = f.bwal.Write(b4[:]); err != nil {
return
}
if _, err = f.bwal.Write(b); err != nil {
return
}
if m := (4 + len(b)) % 16; m != 0 {
var pad [15]byte
_, err = f.bwal.Write(pad[:16-m])
}
return
}
// WAL Packet Tags
const (
wpt00Header = iota
wpt00WriteData
wpt00Checkpoint
wpt00Empty
)
const (
walTypeACIDFiler0 = iota
)
// ACIDFiler0 is a very simple, synchronous implementation of 2PC. It uses a
// single write ahead log file to provide the structural atomicity
// (BeginUpdate/EndUpdate/Rollback) and durability (DB can be recovered from
// WAL if a crash occurred).
//
// ACIDFiler0 is a Filer.
//
// NOTE: Durable synchronous 2PC involves three fsyncs in this implementation
// (WAL, DB, zero truncated WAL). Where possible, it's recommended to collect
// transactions for, say one second before performing the two phase commit as
// the typical performance for rotational hard disks is about few tens of
// fsyncs per second atmost. For an example of such collective transaction
// approach please see the colecting FSM STT in Dbm's documentation[1].
//
// [1]: http://godoc.org/github.com/cznic/exp/dbm
type ACIDFiler0 struct {
*RollbackFiler
bwal *bufio.Writer
data []acidWrite
newEpoch bool
peakWal int64 // tracks WAL maximum used size
testHook bool // keeps WAL untruncated (once)
wal *os.File
walOptions walOptions
}
type walOptions struct {
headroom int64 // Minimum WAL size.
}
// WALOption amends WAL properties.
type WALOption func(*walOptions) error
// MinWAL sets the minimum size a WAL file will have. The "extra" allocated
// file space serves as a headroom. Commits that fit into the headroom should
// not fail due to 'not enough space on the volume' errors.
//
// The min parameter is first rounded-up to a non negative multiple of the size
// of the Allocator atom.
//
// Note: Setting minimum WAL size may render the DB non-recoverable when a
// crash occurs and the DB is opened in an earlier version of LLDB that does
// not support minimum WAL sizes.
func MinWAL(min int64) WALOption {
min = mathutil.MaxInt64(0, min)
if r := min % 16; r != 0 {
min += 16 - r
}
return func(o *walOptions) error {
o.headroom = min
return nil
}
}
// NewACIDFiler0 returns a newly created ACIDFiler0 with WAL in wal.
//
// If the WAL is zero sized then a previous clean shutdown of db is taken for
// granted and no recovery procedure is taken.
//
// If the WAL is of non zero size then it is checked for having a
// committed/fully finished transaction not yet been reflected in db. If such
// transaction exists it's committed to db. If the recovery process finishes
// successfully, the WAL is truncated to the minimum WAL size and fsync'ed
// prior to return from NewACIDFiler0.
//
// opts allow to amend WAL properties.
func NewACIDFiler(db Filer, wal *os.File, opts ...WALOption) (r *ACIDFiler0, err error) {
fi, err := wal.Stat()
if err != nil {
return
}
r = &ACIDFiler0{wal: wal}
for _, o := range opts {
if err := o(&r.walOptions); err != nil {
return nil, err
}
}
if fi.Size() != 0 {
if err = r.recoverDb(db); err != nil {
return
}
}
r.bwal = bufio.NewWriter(r.wal)
r.newEpoch = true
acidWriter := (*acidWriter0)(r)
if r.RollbackFiler, err = NewRollbackFiler(
db,
func(sz int64) (err error) {
// Checkpoint
if err = acidWriter.writePacket([]interface{}{wpt00Checkpoint, sz}); err != nil {
return
}
if err = r.bwal.Flush(); err != nil {
return
}
if err = r.wal.Sync(); err != nil {
return
}
var wfi os.FileInfo
if wfi, err = r.wal.Stat(); err != nil {
return
}
r.peakWal = mathutil.MaxInt64(wfi.Size(), r.peakWal)
// Phase 1 commit complete
for _, v := range r.data {
n := len(v.b)
if m := v.off + int64(n); m > sz {
if n -= int(m - sz); n <= 0 {
continue
}
}
if _, err = db.WriteAt(v.b[:n], v.off); err != nil {
return err
}
}
if err = db.Truncate(sz); err != nil {
return
}
if err = db.Sync(); err != nil {
return
}
// Phase 2 commit complete
if !r.testHook {
if err := r.emptyWAL(); err != nil {
return err
}
}
r.testHook = false
r.bwal.Reset(r.wal)
r.newEpoch = true
return r.wal.Sync()
},
acidWriter,
); err != nil {
return
}
return r, nil
}
func (a *ACIDFiler0) emptyWAL() error {
if err := a.wal.Truncate(a.walOptions.headroom); err != nil {
return err
}
if _, err := a.wal.Seek(0, 0); err != nil {
return err
}
if a.walOptions.headroom != 0 {
a.bwal.Reset(a.wal)
if err := (*acidWriter0)(a).writePacket([]interface{}{wpt00Empty}); err != nil {
return err
}
if err := a.bwal.Flush(); err != nil {
return err
}
if _, err := a.wal.Seek(0, 0); err != nil {
return err
}
}
return nil
}
// PeakWALSize reports the maximum size WAL has ever used.
func (a ACIDFiler0) PeakWALSize() int64 {
return a.peakWal
}
func (a *ACIDFiler0) readPacket(f *bufio.Reader) (items []interface{}, err error) {
var b4 [4]byte
n, err := io.ReadAtLeast(f, b4[:], 4)
if n != 4 {
return
}
ln := int(binary.BigEndian.Uint32(b4[:]))
m := (4 + ln) % 16
padd := (16 - m) % 16
b := make([]byte, ln+padd)
if n, err = io.ReadAtLeast(f, b, len(b)); n != len(b) {
return
}
return DecodeScalars(b[:ln])
}
func (a *ACIDFiler0) recoverDb(db Filer) (err error) {
fi, err := a.wal.Stat()
if err != nil {
return &ErrILSEQ{Type: ErrInvalidWAL, Name: a.wal.Name(), More: err}
}
if sz := fi.Size(); sz%16 != 0 {
return &ErrILSEQ{Type: ErrFileSize, Name: a.wal.Name(), Arg: sz}
}
f := bufio.NewReader(a.wal)
items, err := a.readPacket(f)
if err != nil {
return
}
if items[0] == int64(wpt00Empty) {
if len(items) != 1 {
return &ErrILSEQ{Type: ErrInvalidWAL, Name: a.wal.Name(), More: fmt.Sprintf("invalid packet items %#v", items)}
}
return nil
}
if len(items) != 3 || items[0] != int64(wpt00Header) || items[1] != int64(walTypeACIDFiler0) {
return &ErrILSEQ{Type: ErrInvalidWAL, Name: a.wal.Name(), More: fmt.Sprintf("invalid packet items %#v", items)}
}
tr := NewBTree(nil)
for {
items, err = a.readPacket(f)
if err != nil {
return
}
if len(items) < 2 {
return &ErrILSEQ{Type: ErrInvalidWAL, Name: a.wal.Name(), More: fmt.Sprintf("too few packet items %#v", items)}
}
switch items[0] {
case int64(wpt00WriteData):
if len(items) != 3 {
return &ErrILSEQ{Type: ErrInvalidWAL, Name: a.wal.Name(), More: fmt.Sprintf("invalid data packet items %#v", items)}
}
b, off := items[1].([]byte), items[2].(int64)
var key [8]byte
binary.BigEndian.PutUint64(key[:], uint64(off))
if err = tr.Set(key[:], b); err != nil {
return
}
case int64(wpt00Checkpoint):
var b1 [1]byte
if n, err := f.Read(b1[:]); n != 0 || err == nil {
return &ErrILSEQ{Type: ErrInvalidWAL, Name: a.wal.Name(), More: fmt.Sprintf("checkpoint n %d, err %v", n, err)}
}
if len(items) != 2 {
return &ErrILSEQ{Type: ErrInvalidWAL, Name: a.wal.Name(), More: fmt.Sprintf("checkpoint packet invalid items %#v", items)}
}
sz := items[1].(int64)
enum, err := tr.seekFirst()
if err != nil {
return err
}
for {
var k, v []byte
k, v, err = enum.current()
if err != nil {
if fileutil.IsEOF(err) {
break
}
return err
}
if _, err = db.WriteAt(v, int64(binary.BigEndian.Uint64(k))); err != nil {
return err
}
if err = enum.next(); err != nil {
if fileutil.IsEOF(err) {
break
}
return err
}
}
if err = db.Truncate(sz); err != nil {
return err
}
if err = db.Sync(); err != nil {
return err
}
// Recovery complete
if err := a.emptyWAL(); err != nil {
return err
}
return a.wal.Sync()
default:
return &ErrILSEQ{Type: ErrInvalidWAL, Name: a.wal.Name(), More: fmt.Sprintf("packet tag %v", items[0])}
}
}
}

48
vendor/github.com/cznic/lldb/2pc_docs.go generated vendored
View File

@ -1,48 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
/*
Anatomy of a WAL file
WAL file
A sequence of packets
WAL packet, parts in slice notation
[0:4], 4 bytes: N uint32 // network byte order
[4:4+N], N bytes: payload []byte // gb encoded scalars
Packets, including the 4 byte 'size' prefix, MUST BE padded to size == 0 (mod
16). The values of the padding bytes MUST BE zero.
Encoded scalars first item is a packet type number (packet tag). The meaning of
any other item(s) of the payload depends on the packet tag.
Packet definitions
{wpt00Header int, typ int, s string}
typ: Must be zero (ACIDFiler0 file).
s: Any comment string, empty string is okay.
This packet must be present only once - as the first packet of
a WAL file.
{wpt00WriteData int, b []byte, off int64}
Write data (WriteAt(b, off)).
{wpt00Checkpoint int, sz int64}
Checkpoint (Truncate(sz)).
This packet must be present only once - as the last packet of
a WAL file.
{wpt00Empty int}
The WAL size is of non-zero size due to configured headroom,
but empty otherwise.
*/
package lldb
//TODO optimize bitfiler/wal/2pc data above final size

11
vendor/github.com/cznic/lldb/AUTHORS generated vendored
View File

@ -1,11 +0,0 @@
# This file lists authors for copyright purposes. This file is distinct from
# the CONTRIBUTORS files. See the latter for an explanation.
#
# Names should be added to this file as:
# Name or Organization <email address>
#
# The email address is not required for organizations.
#
# Please keep the list sorted.
Jan Mercl <0xjnml@gmail.com>

10
vendor/github.com/cznic/lldb/CONTRIBUTORS generated vendored
View File

@ -1,10 +0,0 @@
# This file lists people who contributed code to this repository. The AUTHORS
# file lists the copyright holders; this file lists people.
#
# Names should be added to this file like so:
# Name <email address>
#
# Please keep the list sorted.
Jan Mercl <0xjnml@gmail.com>
Patrick Mézard <patrick@mezard.eu>

27
vendor/github.com/cznic/lldb/LICENSE generated vendored
View File

@ -1,27 +0,0 @@
Copyright (c) 2014 The lldb Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the names of the authors nor the names of the
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

51
vendor/github.com/cznic/lldb/Makefile generated vendored
View File

@ -1,51 +0,0 @@
.PHONY: all clean cover cpu editor internalError later mem nuke todo edit
grep=--include=*.go --include=*.l --include=*.y --include=*.yy
ngrep='TODOOK\|parser\.go\|scanner\.go\|.*_string\.go'
all: editor
go vet 2>&1 | grep -v $(ngrep) || true
golint 2>&1 | grep -v $(ngrep) || true
make todo
unused . || true
misspell *.go
gosimple || true
clean:
go clean
rm -f *~ *.test *.out
cover:
t=$(shell tempfile) ; go test -coverprofile $$t && go tool cover -html $$t && unlink $$t
cpu: clean
go test -run @ -bench . -cpuprofile cpu.out
go tool pprof -lines *.test cpu.out
edit:
@2>/dev/null gvim -p Makefile *.go
editor:
gofmt -l -s -w *.go
go test 2>&1 | tee log
go build
internalError:
egrep -ho '"internal error.*"' *.go | sort | cat -n
later:
@grep -n $(grep) LATER * || true
@grep -n $(grep) MAYBE * || true
mem: clean
go test -run @ -bench . -memprofile mem.out -memprofilerate 1 -timeout 24h
go tool pprof -lines -web -alloc_space *.test mem.out
nuke: clean
go clean -i
todo:
@grep -nr $(grep) ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* * || true
@grep -nr $(grep) TODO * || true
@grep -nr $(grep) BUG * || true
@grep -nr $(grep) [^[:alpha:]]println * || true

8
vendor/github.com/cznic/lldb/README.md generated vendored
View File

@ -1,8 +0,0 @@
lldb
====
Package lldb implements a low level database engine.
Installation: $ go get github.com/cznic/lldb
Documentation: [godoc.org/github.com/cznic/lldb](http://godoc.org/github.com/cznic/lldb)

2346
vendor/github.com/cznic/lldb/btree.go generated vendored
View File

File diff suppressed because it is too large Load Diff

170
vendor/github.com/cznic/lldb/errors.go generated vendored
View File

@ -1,170 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Some errors returned by this package.
//
// Note that this package can return more errors than declared here, for
// example io.EOF from Filer.ReadAt().
package lldb
import (
"fmt"
)
// ErrDecodeScalars is possibly returned from DecodeScalars
type ErrDecodeScalars struct {
B []byte // Data being decoded
I int // offending offset
}
// Error implements the built in error type.
func (e *ErrDecodeScalars) Error() string {
return fmt.Sprintf("DecodeScalars: corrupted data @ %d/%d", e.I, len(e.B))
}
// ErrINVAL reports invalid values passed as parameters, for example negative
// offsets where only non-negative ones are allowed or read from the DB.
type ErrINVAL struct {
Src string
Val interface{}
}
// Error implements the built in error type.
func (e *ErrINVAL) Error() string {
return fmt.Sprintf("%s: %+v", e.Src, e.Val)
}
// ErrPERM is for example reported when a Filer is closed while BeginUpdate(s)
// are not balanced with EndUpdate(s)/Rollback(s) or when EndUpdate or Rollback
// is invoked which is not paired with a BeginUpdate.
type ErrPERM struct {
Src string
}
// Error implements the built in error type.
func (e *ErrPERM) Error() string {
return fmt.Sprintf("%s: Operation not permitted", e.Src)
}
// ErrTag represents an ErrILSEQ kind.
type ErrType int
// ErrILSEQ types
const (
ErrOther ErrType = iota
ErrAdjacentFree // Adjacent free blocks (.Off and .Arg)
ErrDecompress // Used compressed block: corrupted compression
ErrExpFreeTag // Expected a free block tag, got .Arg
ErrExpUsedTag // Expected a used block tag, got .Arg
ErrFLT // Free block is invalid or referenced multiple times
ErrFLTLoad // FLT truncated to .Off, need size >= .Arg
ErrFLTSize // Free block size (.Arg) doesn't belong to its list min size: .Arg2
ErrFileSize // File .Name size (.Arg) != 0 (mod 16)
ErrFreeChaining // Free block, .prev.next doesn't point back to this block
ErrFreeTailBlock // Last block is free
ErrHead // Head of a free block list has non zero Prev (.Arg)
ErrInvalidRelocTarget // Reloc doesn't target (.Arg) a short or long used block
ErrInvalidWAL // Corrupted write ahead log. .Name: file name, .More: more
ErrLongFreeBlkTooLong // Long free block spans beyond EOF, size .Arg
ErrLongFreeBlkTooShort // Long free block must have at least 2 atoms, got only .Arg
ErrLongFreeNextBeyondEOF // Long free block .Next (.Arg) spans beyond EOF
ErrLongFreePrevBeyondEOF // Long free block .Prev (.Arg) spans beyond EOF
ErrLongFreeTailTag // Expected a long free block tail tag, got .Arg
ErrLostFreeBlock // Free block is not in any FLT list
ErrNullReloc // Used reloc block with nil target
ErrRelocBeyondEOF // Used reloc points (.Arg) beyond EOF
ErrShortFreeTailTag // Expected a short free block tail tag, got .Arg
ErrSmall // Request for a free block (.Arg) returned a too small one (.Arg2) at .Off
ErrTailTag // Block at .Off has invalid tail CC (compression code) tag, got .Arg
ErrUnexpReloc // Unexpected reloc block referred to from reloc block .Arg
ErrVerifyPadding // Used block has nonzero padding
ErrVerifyTailSize // Long free block size .Arg but tail size .Arg2
ErrVerifyUsedSpan // Used block size (.Arg) spans beyond EOF
)
// ErrILSEQ reports a corrupted file format. Details in fields according to Type.
type ErrILSEQ struct {
Type ErrType
Off int64
Arg int64
Arg2 int64
Arg3 int64
Name string
More interface{}
}
// Error implements the built in error type.
func (e *ErrILSEQ) Error() string {
switch e.Type {
case ErrAdjacentFree:
return fmt.Sprintf("Adjacent free blocks at offset %#x and %#x", e.Off, e.Arg)
case ErrDecompress:
return fmt.Sprintf("Compressed block at offset %#x: Corrupted compressed content", e.Off)
case ErrExpFreeTag:
return fmt.Sprintf("Block at offset %#x: Expected a free block tag, got %#2x", e.Off, e.Arg)
case ErrExpUsedTag:
return fmt.Sprintf("Block at ofset %#x: Expected a used block tag, got %#2x", e.Off, e.Arg)
case ErrFLT:
return fmt.Sprintf("Free block at offset %#x is invalid or referenced multiple times", e.Off)
case ErrFLTLoad:
return fmt.Sprintf("FLT truncated to size %d, expected at least %d", e.Off, e.Arg)
case ErrFLTSize:
return fmt.Sprintf("Free block at offset %#x has size (%#x) should be at least (%#x)", e.Off, e.Arg, e.Arg2)
case ErrFileSize:
return fmt.Sprintf("File %q size (%#x) != 0 (mod 16)", e.Name, e.Arg)
case ErrFreeChaining:
return fmt.Sprintf("Free block at offset %#x: .prev.next doesn point back here.", e.Off)
case ErrFreeTailBlock:
return fmt.Sprintf("Free block at offset %#x: Cannot be last file block", e.Off)
case ErrHead:
return fmt.Sprintf("Block at offset %#x: Head of free block list has non zero .prev %#x", e.Off, e.Arg)
case ErrInvalidRelocTarget:
return fmt.Sprintf("Used reloc block at offset %#x: Target (%#x) is not a short or long used block", e.Off, e.Arg)
case ErrInvalidWAL:
return fmt.Sprintf("Corrupted write ahead log file: %q %v", e.Name, e.More)
case ErrLongFreeBlkTooLong:
return fmt.Sprintf("Long free block at offset %#x: Size (%#x) beyond EOF", e.Off, e.Arg)
case ErrLongFreeBlkTooShort:
return fmt.Sprintf("Long free block at offset %#x: Size (%#x) too small", e.Off, e.Arg)
case ErrLongFreeNextBeyondEOF:
return fmt.Sprintf("Long free block at offset %#x: Next (%#x) points beyond EOF", e.Off, e.Arg)
case ErrLongFreePrevBeyondEOF:
return fmt.Sprintf("Long free block at offset %#x: Prev (%#x) points beyond EOF", e.Off, e.Arg)
case ErrLongFreeTailTag:
return fmt.Sprintf("Block at offset %#x: Expected long free tail tag, got %#2x", e.Off, e.Arg)
case ErrLostFreeBlock:
return fmt.Sprintf("Free block at offset %#x: not in any FLT list", e.Off)
case ErrNullReloc:
return fmt.Sprintf("Used reloc block at offset %#x: Nil target", e.Off)
case ErrRelocBeyondEOF:
return fmt.Sprintf("Used reloc block at offset %#x: Link (%#x) points beyond EOF", e.Off, e.Arg)
case ErrShortFreeTailTag:
return fmt.Sprintf("Block at offset %#x: Expected short free tail tag, got %#2x", e.Off, e.Arg)
case ErrSmall:
return fmt.Sprintf("Request for of free block of size %d returned a too small (%d) one at offset %#x", e.Arg, e.Arg2, e.Off)
case ErrTailTag:
return fmt.Sprintf("Block at offset %#x: Invalid tail CC tag, got %#2x", e.Off, e.Arg)
case ErrUnexpReloc:
return fmt.Sprintf("Block at offset %#x: Unexpected reloc block. Referred to from reloc block at offset %#x", e.Off, e.Arg)
case ErrVerifyPadding:
return fmt.Sprintf("Used block at offset %#x: Nonzero padding", e.Off)
case ErrVerifyTailSize:
return fmt.Sprintf("Long free block at offset %#x: Size %#x, but tail size %#x", e.Off, e.Arg, e.Arg2)
case ErrVerifyUsedSpan:
return fmt.Sprintf("Used block at offset %#x: Size %#x spans beyond EOF", e.Off, e.Arg)
}
more := ""
if e.More != nil {
more = fmt.Sprintf(", %v", e.More)
}
off := ""
if e.Off != 0 {
off = fmt.Sprintf(", off: %#x", e.Off)
}
return fmt.Sprintf("Error%s%s", off, more)
}

1999
vendor/github.com/cznic/lldb/falloc.go generated vendored
View File

File diff suppressed because it is too large Load Diff

184
vendor/github.com/cznic/lldb/filer.go generated vendored
View File

@ -1,184 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// An abstraction of file like (persistent) storage with optional (abstracted)
// support for structural integrity.
package lldb
import "github.com/cznic/mathutil"
// A Filer is a []byte-like model of a file or similar entity. It may
// optionally implement support for structural transaction safety. In contrast
// to a file stream, a Filer is not sequentially accessible. ReadAt and WriteAt
// are always "addressed" by an offset and are assumed to perform atomically.
// A Filer is not safe for concurrent access, it's designed for consumption by
// the other objects in package, which should use a Filer from one goroutine
// only or via a mutex. BeginUpdate, EndUpdate and Rollback must be either all
// implemented by a Filer for structural integrity - or they should be all
// no-ops; where/if that requirement is relaxed.
//
// If a Filer wraps another Filer implementation, it usually invokes the same
// methods on the "inner" one, after some possible argument translations etc.
// If a Filer implements the structural transactions handling methods
// (BeginUpdate, EndUpdate and Rollback) as no-ops _and_ wraps another Filer:
// it then still MUST invoke those methods on the inner Filer. This is
// important for the case where a RollbackFiler exists somewhere down the
// chain. It's also important for an Allocator - to know when it must
// invalidate its FLT cache.
type Filer interface {
// BeginUpdate increments the "nesting" counter (initially zero). Every
// call to BeginUpdate must be eventually "balanced" by exactly one of
// EndUpdate or Rollback. Calls to BeginUpdate may nest.
BeginUpdate() error
// Analogous to os.File.Close().
Close() error
// EndUpdate decrements the "nesting" counter. If it's zero after that
// then assume the "storage" has reached structural integrity (after a
// batch of partial updates). If a Filer implements some support for
// that (write ahead log, journal, etc.) then the appropriate actions
// are to be taken for nesting == 0. Invocation of an unbalanced
// EndUpdate is an error.
EndUpdate() error
// Analogous to os.File.Name().
Name() string
// PunchHole deallocates space inside a "file" in the byte range
// starting at off and continuing for size bytes. The actual hole
// created by PunchHole may be smaller than requested. The Filer size
// (as reported by `Size()` does not change when hole punching, even
// when punching the end of a file off. In contrast to the Linux
// implementation of FALLOC_FL_PUNCH_HOLE in `fallocate`(2); a Filer is
// free not only to ignore `PunchHole()` (implement it as a nop), but
// additionally no guarantees about the content of the hole, when
// eventually read back, are required, i.e. any data, not only zeros,
// can be read from the "hole", including just anything what was left
// there - with all of the possible security problems.
PunchHole(off, size int64) error
// As os.File.ReadAt. Note: `off` is an absolute "file pointer"
// address and cannot be negative even when a Filer is a InnerFiler.
ReadAt(b []byte, off int64) (n int, err error)
// Rollback cancels and undoes the innermost pending update level.
// Rollback decrements the "nesting" counter. If a Filer implements
// some support for keeping structural integrity (write ahead log,
// journal, etc.) then the appropriate actions are to be taken.
// Invocation of an unbalanced Rollback is an error.
Rollback() error
// Analogous to os.File.FileInfo().Size().
Size() (int64, error)
// Analogous to os.Sync().
Sync() (err error)
// Analogous to os.File.Truncate().
Truncate(size int64) error
// Analogous to os.File.WriteAt(). Note: `off` is an absolute "file
// pointer" address and cannot be negative even when a Filer is a
// InnerFiler.
WriteAt(b []byte, off int64) (n int, err error)
}
var _ Filer = &InnerFiler{} // Ensure InnerFiler is a Filer.
// A InnerFiler is a Filer with added addressing/size translation.
type InnerFiler struct {
outer Filer
off int64
}
// NewInnerFiler returns a new InnerFiler wrapped by `outer` in a way which
// adds `off` to every access.
//
// For example, considering:
//
// inner := NewInnerFiler(outer, 10)
//
// then
//
// inner.WriteAt([]byte{42}, 4)
//
// translates to
//
// outer.WriteAt([]byte{42}, 14)
//
// But an attempt to emulate
//
// outer.WriteAt([]byte{17}, 9)
//
// by
//
// inner.WriteAt([]byte{17}, -1)
//
// will fail as the `off` parameter can never be < 0. Also note that
//
// inner.Size() == outer.Size() - off,
//
// i.e. `inner` pretends no `outer` exists. Finally, after e.g.
//
// inner.Truncate(7)
// outer.Size() == 17
//
// will be true.
func NewInnerFiler(outer Filer, off int64) *InnerFiler { return &InnerFiler{outer, off} }
// BeginUpdate implements Filer.
func (f *InnerFiler) BeginUpdate() error { return f.outer.BeginUpdate() }
// Close implements Filer.
func (f *InnerFiler) Close() (err error) { return f.outer.Close() }
// EndUpdate implements Filer.
func (f *InnerFiler) EndUpdate() error { return f.outer.EndUpdate() }
// Name implements Filer.
func (f *InnerFiler) Name() string { return f.outer.Name() }
// PunchHole implements Filer. `off`, `size` must be >= 0.
func (f *InnerFiler) PunchHole(off, size int64) error { return f.outer.PunchHole(f.off+off, size) }
// ReadAt implements Filer. `off` must be >= 0.
func (f *InnerFiler) ReadAt(b []byte, off int64) (n int, err error) {
if off < 0 {
return 0, &ErrINVAL{f.outer.Name() + ":ReadAt invalid off", off}
}
return f.outer.ReadAt(b, f.off+off)
}
// Rollback implements Filer.
func (f *InnerFiler) Rollback() error { return f.outer.Rollback() }
// Size implements Filer.
func (f *InnerFiler) Size() (int64, error) {
sz, err := f.outer.Size()
if err != nil {
return 0, err
}
return mathutil.MaxInt64(sz-f.off, 0), nil
}
// Sync() implements Filer.
func (f *InnerFiler) Sync() (err error) {
return f.outer.Sync()
}
// Truncate implements Filer.
func (f *InnerFiler) Truncate(size int64) error { return f.outer.Truncate(size + f.off) }
// WriteAt implements Filer. `off` must be >= 0.
func (f *InnerFiler) WriteAt(b []byte, off int64) (n int, err error) {
if off < 0 {
return 0, &ErrINVAL{f.outer.Name() + ":WriteAt invalid off", off}
}
return f.outer.WriteAt(b, f.off+off)
}

812
vendor/github.com/cznic/lldb/gb.go generated vendored
View File

@ -1,812 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Utilities to encode/decode and collate Go predeclared scalar types (and the
// typeless nil and []byte). The encoding format is a variation of the one
// used by the "encoding/gob" package.
package lldb
import (
"bytes"
"fmt"
"math"
"github.com/cznic/mathutil"
)
const (
gbNull = iota // 0x00
gbFalse // 0x01
gbTrue // 0x02
gbFloat0 // 0x03
gbFloat1 // 0x04
gbFloat2 // 0x05
gbFloat3 // 0x06
gbFloat4 // 0x07
gbFloat5 // 0x08
gbFloat6 // 0x09
gbFloat7 // 0x0a
gbFloat8 // 0x0b
gbComplex0 // 0x0c
gbComplex1 // 0x0d
gbComplex2 // 0x0e
gbComplex3 // 0x0f
gbComplex4 // 0x10
gbComplex5 // 0x11
gbComplex6 // 0x12
gbComplex7 // 0x13
gbComplex8 // 0x14
gbBytes00 // 0x15
gbBytes01 // 0x16
gbBytes02 // 0x17
gbBytes03 // 0x18
gbBytes04 // 0x19
gbBytes05 // 0x1a
gbBytes06 // 0x1b
gbBytes07 // 0x1c
gbBytes08 // 0x1d
gbBytes09 // 0x1e
gbBytes10 // 0x1f
gbBytes11 // 0x20
gbBytes12 // 0x21
gbBytes13 // 0x22
gbBytes14 // 0x23
gbBytes15 // 0x24
gbBytes16 // 0x25
gbBytes17 // Ox26
gbBytes1 // 0x27
gbBytes2 // 0x28: Offset by one to allow 64kB sized []byte.
gbString00 // 0x29
gbString01 // 0x2a
gbString02 // 0x2b
gbString03 // 0x2c
gbString04 // 0x2d
gbString05 // 0x2e
gbString06 // 0x2f
gbString07 // 0x30
gbString08 // 0x31
gbString09 // 0x32
gbString10 // 0x33
gbString11 // 0x34
gbString12 // 0x35
gbString13 // 0x36
gbString14 // 0x37
gbString15 // 0x38
gbString16 // 0x39
gbString17 // 0x3a
gbString1 // 0x3b
gbString2 // 0x3c
gbUintP1 // 0x3d
gbUintP2 // 0x3e
gbUintP3 // 0x3f
gbUintP4 // 0x40
gbUintP5 // 0x41
gbUintP6 // 0x42
gbUintP7 // 0x43
gbUintP8 // 0x44
gbIntM8 // 0x45
gbIntM7 // 0x46
gbIntM6 // 0x47
gbIntM5 // 0x48
gbIntM4 // 0x49
gbIntM3 // 0x4a
gbIntM2 // 0x4b
gbIntM1 // 0x4c
gbIntP1 // 0x4d
gbIntP2 // 0x4e
gbIntP3 // 0x4f
gbIntP4 // 0x50
gbIntP5 // 0x51
gbIntP6 // 0x52
gbIntP7 // 0x53
gbIntP8 // 0x54
gbInt0 // 0x55
gbIntMax = 255 - gbInt0 // 0xff == 170
)
// EncodeScalars encodes a vector of predeclared scalar type values to a
// []byte, making it suitable to store it as a "record" in a DB or to use it as
// a key of a BTree.
func EncodeScalars(scalars ...interface{}) (b []byte, err error) {
for _, scalar := range scalars {
switch x := scalar.(type) {
default:
return nil, &ErrINVAL{"EncodeScalars: unsupported type", fmt.Sprintf("%T in `%#v`", x, scalars)}
case nil:
b = append(b, gbNull)
case bool:
switch x {
case false:
b = append(b, gbFalse)
case true:
b = append(b, gbTrue)
}
case float32:
encFloat(float64(x), &b)
case float64:
encFloat(x, &b)
case complex64:
encComplex(complex128(x), &b)
case complex128:
encComplex(x, &b)
case string:
n := len(x)
if n <= 17 {
b = append(b, byte(gbString00+n))
b = append(b, []byte(x)...)
break
}
if n > 65535 {
return nil, fmt.Errorf("EncodeScalars: cannot encode string of length %d (limit 65536)", n)
}
pref := byte(gbString1)
if n > 255 {
pref++
}
b = append(b, pref)
encUint0(uint64(n), &b)
b = append(b, []byte(x)...)
case int8:
encInt(int64(x), &b)
case int16:
encInt(int64(x), &b)
case int32:
encInt(int64(x), &b)
case int64:
encInt(x, &b)
case int:
encInt(int64(x), &b)
case uint8:
encUint(uint64(x), &b)
case uint16:
encUint(uint64(x), &b)
case uint32:
encUint(uint64(x), &b)
case uint64:
encUint(x, &b)
case uint:
encUint(uint64(x), &b)
case []byte:
n := len(x)
if n <= 17 {
b = append(b, byte(gbBytes00+n))
b = append(b, x...)
break
}
if n > 655356 {
return nil, fmt.Errorf("EncodeScalars: cannot encode []byte of length %d (limit 65536)", n)
}
pref := byte(gbBytes1)
if n > 255 {
pref++
}
b = append(b, pref)
if n <= 255 {
b = append(b, byte(n))
} else {
n--
b = append(b, byte(n>>8), byte(n))
}
b = append(b, x...)
}
}
return
}
func encComplex(f complex128, b *[]byte) {
encFloatPrefix(gbComplex0, real(f), b)
encFloatPrefix(gbComplex0, imag(f), b)
}
func encFloatPrefix(prefix byte, f float64, b *[]byte) {
u := math.Float64bits(f)
var n uint64
for i := 0; i < 8; i++ {
n <<= 8
n |= u & 0xFF
u >>= 8
}
bits := mathutil.BitLenUint64(n)
if bits == 0 {
*b = append(*b, prefix)
return
}
// 0 1 2 3 4 5 6 7 8 9
// . 1 1 1 1 1 1 1 1 2
encUintPrefix(prefix+1+byte((bits-1)>>3), n, b)
}
func encFloat(f float64, b *[]byte) {
encFloatPrefix(gbFloat0, f, b)
}
func encUint0(n uint64, b *[]byte) {
switch {
case n <= 0xff:
*b = append(*b, byte(n))
case n <= 0xffff:
*b = append(*b, byte(n>>8), byte(n))
case n <= 0xffffff:
*b = append(*b, byte(n>>16), byte(n>>8), byte(n))
case n <= 0xffffffff:
*b = append(*b, byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n <= 0xffffffffff:
*b = append(*b, byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n <= 0xffffffffffff:
*b = append(*b, byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n <= 0xffffffffffffff:
*b = append(*b, byte(n>>48), byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n <= math.MaxUint64:
*b = append(*b, byte(n>>56), byte(n>>48), byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
}
}
func encUintPrefix(prefix byte, n uint64, b *[]byte) {
*b = append(*b, prefix)
encUint0(n, b)
}
func encUint(n uint64, b *[]byte) {
bits := mathutil.Max(1, mathutil.BitLenUint64(n))
encUintPrefix(gbUintP1+byte((bits-1)>>3), n, b)
}
func encInt(n int64, b *[]byte) {
switch {
case n < -0x100000000000000:
*b = append(*b, byte(gbIntM8), byte(n>>56), byte(n>>48), byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n < -0x1000000000000:
*b = append(*b, byte(gbIntM7), byte(n>>48), byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n < -0x10000000000:
*b = append(*b, byte(gbIntM6), byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n < -0x100000000:
*b = append(*b, byte(gbIntM5), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n < -0x1000000:
*b = append(*b, byte(gbIntM4), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n < -0x10000:
*b = append(*b, byte(gbIntM3), byte(n>>16), byte(n>>8), byte(n))
case n < -0x100:
*b = append(*b, byte(gbIntM2), byte(n>>8), byte(n))
case n < 0:
*b = append(*b, byte(gbIntM1), byte(n))
case n <= gbIntMax:
*b = append(*b, byte(gbInt0+n))
case n <= 0xff:
*b = append(*b, gbIntP1, byte(n))
case n <= 0xffff:
*b = append(*b, gbIntP2, byte(n>>8), byte(n))
case n <= 0xffffff:
*b = append(*b, gbIntP3, byte(n>>16), byte(n>>8), byte(n))
case n <= 0xffffffff:
*b = append(*b, gbIntP4, byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n <= 0xffffffffff:
*b = append(*b, gbIntP5, byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n <= 0xffffffffffff:
*b = append(*b, gbIntP6, byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n <= 0xffffffffffffff:
*b = append(*b, gbIntP7, byte(n>>48), byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
case n <= 0x7fffffffffffffff:
*b = append(*b, gbIntP8, byte(n>>56), byte(n>>48), byte(n>>40), byte(n>>32), byte(n>>24), byte(n>>16), byte(n>>8), byte(n))
}
}
func decodeFloat(b []byte) float64 {
var u uint64
for i, v := range b {
u |= uint64(v) << uint((i+8-len(b))*8)
}
return math.Float64frombits(u)
}
// DecodeScalars decodes a []byte produced by EncodeScalars.
func DecodeScalars(b []byte) (scalars []interface{}, err error) {
b0 := b
for len(b) != 0 {
switch tag := b[0]; tag {
//default:
//return nil, fmt.Errorf("tag %d(%#x) not supported", b[0], b[0])
case gbNull:
scalars = append(scalars, nil)
b = b[1:]
case gbFalse:
scalars = append(scalars, false)
b = b[1:]
case gbTrue:
scalars = append(scalars, true)
b = b[1:]
case gbFloat0:
scalars = append(scalars, 0.0)
b = b[1:]
case gbFloat1, gbFloat2, gbFloat3, gbFloat4, gbFloat5, gbFloat6, gbFloat7, gbFloat8:
n := 1 + int(tag) - gbFloat0
if len(b) < n-1 {
goto corrupted
}
scalars = append(scalars, decodeFloat(b[1:n]))
b = b[n:]
case gbComplex0, gbComplex1, gbComplex2, gbComplex3, gbComplex4, gbComplex5, gbComplex6, gbComplex7, gbComplex8:
n := 1 + int(tag) - gbComplex0
if len(b) < n-1 {
goto corrupted
}
re := decodeFloat(b[1:n])
b = b[n:]
if len(b) == 0 {
goto corrupted
}
tag = b[0]
if tag < gbComplex0 || tag > gbComplex8 {
goto corrupted
}
n = 1 + int(tag) - gbComplex0
if len(b) < n-1 {
goto corrupted
}
scalars = append(scalars, complex(re, decodeFloat(b[1:n])))
b = b[n:]
case gbBytes00, gbBytes01, gbBytes02, gbBytes03, gbBytes04,
gbBytes05, gbBytes06, gbBytes07, gbBytes08, gbBytes09,
gbBytes10, gbBytes11, gbBytes12, gbBytes13, gbBytes14,
gbBytes15, gbBytes16, gbBytes17:
n := int(tag - gbBytes00)
if len(b) < n+1 {
goto corrupted
}
scalars = append(scalars, append([]byte(nil), b[1:n+1]...))
b = b[n+1:]
case gbBytes1:
if len(b) < 2 {
goto corrupted
}
n := int(b[1])
b = b[2:]
if len(b) < n {
goto corrupted
}
scalars = append(scalars, append([]byte(nil), b[:n]...))
b = b[n:]
case gbBytes2:
if len(b) < 3 {
goto corrupted
}
n := int(b[1])<<8 | int(b[2]) + 1
b = b[3:]
if len(b) < n {
goto corrupted
}
scalars = append(scalars, append([]byte(nil), b[:n]...))
b = b[n:]
case gbString00, gbString01, gbString02, gbString03, gbString04,
gbString05, gbString06, gbString07, gbString08, gbString09,
gbString10, gbString11, gbString12, gbString13, gbString14,
gbString15, gbString16, gbString17:
n := int(tag - gbString00)
if len(b) < n+1 {
goto corrupted
}
scalars = append(scalars, string(b[1:n+1]))
b = b[n+1:]
case gbString1:
if len(b) < 2 {
goto corrupted
}
n := int(b[1])
b = b[2:]
if len(b) < n {
goto corrupted
}
scalars = append(scalars, string(b[:n]))
b = b[n:]
case gbString2:
if len(b) < 3 {
goto corrupted
}
n := int(b[1])<<8 | int(b[2])
b = b[3:]
if len(b) < n {
goto corrupted
}
scalars = append(scalars, string(b[:n]))
b = b[n:]
case gbUintP1, gbUintP2, gbUintP3, gbUintP4, gbUintP5, gbUintP6, gbUintP7, gbUintP8:
b = b[1:]
n := 1 + int(tag) - gbUintP1
if len(b) < n {
goto corrupted
}
var u uint64
for _, v := range b[:n] {
u = u<<8 | uint64(v)
}
scalars = append(scalars, u)
b = b[n:]
case gbIntM8, gbIntM7, gbIntM6, gbIntM5, gbIntM4, gbIntM3, gbIntM2, gbIntM1:
b = b[1:]
n := 8 - (int(tag) - gbIntM8)
if len(b) < n {
goto corrupted
}
u := uint64(math.MaxUint64)
for _, v := range b[:n] {
u = u<<8 | uint64(v)
}
scalars = append(scalars, int64(u))
b = b[n:]
case gbIntP1, gbIntP2, gbIntP3, gbIntP4, gbIntP5, gbIntP6, gbIntP7, gbIntP8:
b = b[1:]
n := 1 + int(tag) - gbIntP1
if len(b) < n {
goto corrupted
}
i := int64(0)
for _, v := range b[:n] {
i = i<<8 | int64(v)
}
scalars = append(scalars, i)
b = b[n:]
default:
scalars = append(scalars, int64(b[0])-gbInt0)
b = b[1:]
}
}
return append([]interface{}(nil), scalars...), nil
corrupted:
return nil, &ErrDecodeScalars{append([]byte(nil), b0...), len(b0) - len(b)}
}
func collateComplex(x, y complex128) int {
switch rx, ry := real(x), real(y); {
case rx < ry:
return -1
case rx == ry:
switch ix, iy := imag(x), imag(y); {
case ix < iy:
return -1
case ix == iy:
return 0
case ix > iy:
return 1
}
}
//case rx > ry:
return 1
}
func collateFloat(x, y float64) int {
switch {
case x < y:
return -1
case x == y:
return 0
}
//case x > y:
return 1
}
func collateInt(x, y int64) int {
switch {
case x < y:
return -1
case x == y:
return 0
}
//case x > y:
return 1
}
func collateUint(x, y uint64) int {
switch {
case x < y:
return -1
case x == y:
return 0
}
//case x > y:
return 1
}
func collateIntUint(x int64, y uint64) int {
if y > math.MaxInt64 {
return -1
}
return collateInt(x, int64(y))
}
func collateUintInt(x uint64, y int64) int {
return -collateIntUint(y, x)
}
func collateType(i interface{}) (r interface{}, err error) {
switch x := i.(type) {
default:
return nil, fmt.Errorf("invalid collate type %T", x)
case nil:
return i, nil
case bool:
return i, nil
case int8:
return int64(x), nil
case int16:
return int64(x), nil
case int32:
return int64(x), nil
case int64:
return i, nil
case int:
return int64(x), nil
case uint8:
return uint64(x), nil
case uint16:
return uint64(x), nil
case uint32:
return uint64(x), nil
case uint64:
return i, nil
case uint:
return uint64(x), nil
case float32:
return float64(x), nil
case float64:
return i, nil
case complex64:
return complex128(x), nil
case complex128:
return i, nil
case []byte:
return i, nil
case string:
return i, nil
}
}
// Collate collates two arrays of Go predeclared scalar types (and the typeless
// nil or []byte). If any other type appears in x or y, Collate will return a
// non nil error. String items are collated using strCollate or lexically
// byte-wise (as when using Go comparison operators) when strCollate is nil.
// []byte items are collated using bytes.Compare.
//
// Collate returns:
//
// -1 if x < y
// 0 if x == y
// +1 if x > y
//
// The same value as defined above must be returned from strCollate.
//
// The "outer" ordering is: nil, bool, number, []byte, string. IOW, nil is
// "smaller" than anything else except other nil, numbers collate before
// []byte, []byte collate before strings, etc.
//
// Integers and real numbers collate as expected in math. However, complex
// numbers are not ordered in Go. Here the ordering is defined: Complex numbers
// are in comparison considered first only by their real part. Iff the result
// is equality then the imaginary part is used to determine the ordering. In
// this "second order" comparing, integers and real numbers are considered as
// complex numbers with a zero imaginary part.
func Collate(x, y []interface{}, strCollate func(string, string) int) (r int, err error) {
nx, ny := len(x), len(y)
switch {
case nx == 0 && ny != 0:
return -1, nil
case nx == 0 && ny == 0:
return 0, nil
case nx != 0 && ny == 0:
return 1, nil
}
r = 1
if nx > ny {
x, y, r = y, x, -r
}
var c int
for i, xi0 := range x {
yi0 := y[i]
xi, err := collateType(xi0)
if err != nil {
return 0, err
}
yi, err := collateType(yi0)
if err != nil {
return 0, err
}
switch x := xi.(type) {
default:
panic(fmt.Errorf("internal error: %T", x))
case nil:
switch yi.(type) {
case nil:
// nop
default:
return -r, nil
}
case bool:
switch y := yi.(type) {
case nil:
return r, nil
case bool:
switch {
case !x && y:
return -r, nil
case x == y:
// nop
case x && !y:
return r, nil
}
default:
return -r, nil
}
case int64:
switch y := yi.(type) {
case nil, bool:
return r, nil
case int64:
c = collateInt(x, y)
case uint64:
c = collateIntUint(x, y)
case float64:
c = collateFloat(float64(x), y)
case complex128:
c = collateComplex(complex(float64(x), 0), y)
case []byte:
return -r, nil
case string:
return -r, nil
}
if c != 0 {
return c * r, nil
}
case uint64:
switch y := yi.(type) {
case nil, bool:
return r, nil
case int64:
c = collateUintInt(x, y)
case uint64:
c = collateUint(x, y)
case float64:
c = collateFloat(float64(x), y)
case complex128:
c = collateComplex(complex(float64(x), 0), y)
case []byte:
return -r, nil
case string:
return -r, nil
}
if c != 0 {
return c * r, nil
}
case float64:
switch y := yi.(type) {
case nil, bool:
return r, nil
case int64:
c = collateFloat(x, float64(y))
case uint64:
c = collateFloat(x, float64(y))
case float64:
c = collateFloat(x, y)
case complex128:
c = collateComplex(complex(x, 0), y)
case []byte:
return -r, nil
case string:
return -r, nil
}
if c != 0 {
return c * r, nil
}
case complex128:
switch y := yi.(type) {
case nil, bool:
return r, nil
case int64:
c = collateComplex(x, complex(float64(y), 0))
case uint64:
c = collateComplex(x, complex(float64(y), 0))
case float64:
c = collateComplex(x, complex(y, 0))
case complex128:
c = collateComplex(x, y)
case []byte:
return -r, nil
case string:
return -r, nil
}
if c != 0 {
return c * r, nil
}
case []byte:
switch y := yi.(type) {
case nil, bool, int64, uint64, float64, complex128:
return r, nil
case []byte:
c = bytes.Compare(x, y)
case string:
return -r, nil
}
if c != 0 {
return c * r, nil
}
case string:
switch y := yi.(type) {
case nil, bool, int64, uint64, float64, complex128:
return r, nil
case []byte:
return r, nil
case string:
switch {
case strCollate != nil:
c = strCollate(x, y)
case x < y:
return -r, nil
case x == y:
c = 0
case x > y:
return r, nil
}
}
if c != 0 {
return c * r, nil
}
}
}
if nx == ny {
return 0, nil
}
return -r, nil
}

160
vendor/github.com/cznic/lldb/lldb.go generated vendored
View File

@ -1,160 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package lldb implements a low level database engine. The database model used
// could be considered a specific implementation of some small(est)
// intersection of models listed in [1]. As a settled term is lacking, it'll be
// called here a 'Virtual memory model' (VMM).
//
// Changelog
//
// 2016-07-24: v1.0.4 brings some performance improvements.
//
// 2016-07-22: v1.0.3 brings some small performance improvements.
//
// 2016-07-12: v1.0.2 now uses packages from cznic/internal.
//
// 2016-07-12: v1.0.1 adds a license for testdata/fortunes.txt.
//
// 2016-07-11: First standalone release v1.0.0 of the package previously
// published as experimental (github.com/cznic/exp/lldb).
//
// Filers
//
// A Filer is an abstraction of storage. A Filer may be a part of some process'
// virtual address space, an OS file, a networked, remote file etc. Persistence
// of the storage is optional, opaque to VMM and it is specific to a concrete
// Filer implementation.
//
// Space management
//
// Mechanism to allocate, reallocate (resize), deallocate (and later reclaim
// the unused) contiguous parts of a Filer, called blocks. Blocks are
// identified and referred to by a handle, an int64.
//
// BTrees
//
// In addition to the VMM like services, lldb provides volatile and
// non-volatile BTrees. Keys and values of a BTree are limited in size to 64kB
// each (a bit more actually). Support for larger keys/values, if desired, can
// be built atop a BTree to certain limits.
//
// Handles vs pointers
//
// A handle is the abstracted storage counterpart of a memory address. There
// is one fundamental difference, though. Resizing a block never results in a
// change to the handle which refers to the resized block, so a handle is more
// akin to an unique numeric id/key. Yet it shares one property of pointers -
// handles can be associated again with blocks after the original handle block
// was deallocated. In other words, a handle uniqueness domain is the state of
// the database and is not something comparable to e.g. an ever growing
// numbering sequence.
//
// Also, as with memory pointers, dangling handles can be created and blocks
// overwritten when such handles are used. Using a zero handle to refer to a
// block will not panic; however, the resulting error is effectively the same
// exceptional situation as dereferencing a nil pointer.
//
// Blocks
//
// Allocated/used blocks, are limited in size to only a little bit more than
// 64kB. Bigger semantic entities/structures must be built in lldb's client
// code. The content of a block has no semantics attached, it's only a fully
// opaque `[]byte`.
//
// Scalars
//
// Use of "scalars" applies to EncodeScalars, DecodeScalars and Collate. Those
// first two "to bytes" and "from bytes" functions are suggested for handling
// multi-valued Allocator content items and/or keys/values of BTrees (using
// Collate for keys). Types called "scalar" are:
//
// nil (the typeless one)
// bool
// all integral types: [u]int8, [u]int16, [u]int32, [u]int, [u]int64
// all floating point types: float32, float64
// all complex types: complex64, complex128
// []byte (64kB max)
// string (64kb max)
//
// Specific implementations
//
// Included are concrete implementations of some of the VMM interfaces included
// to ease serving simple client code or for testing and possibly as an
// example. More details in the documentation of such implementations.
//
// [1]: http://en.wikipedia.org/wiki/Database_model
package lldb
const (
fltSz = 0x70 // size of the FLT
maxShort = 251
maxRq = 65787
maxFLTRq = 4112
maxHandle = 1<<56 - 1
atomLen = 16
tagUsedLong = 0xfc
tagUsedRelocated = 0xfd
tagFreeShort = 0xfe
tagFreeLong = 0xff
tagNotCompressed = 0
tagCompressed = 1
)
// Content size n -> blocksize in atoms.
func n2atoms(n int) int {
if n > maxShort {
n += 2
}
return (n+1)/16 + 1
}
// Content size n -> number of padding zeros.
func n2padding(n int) int {
if n > maxShort {
n += 2
}
return 15 - (n+1)&15
}
// Handle <-> offset
func h2off(h int64) int64 { return (h + 6) * 16 }
func off2h(off int64) int64 { return off/16 - 6 }
// Get a 7B int64 from b
func b2h(b []byte) (h int64) {
for _, v := range b[:7] {
h = h<<8 | int64(v)
}
return
}
// Put a 7B int64 into b
func h2b(b []byte, h int64) []byte {
for i := range b[:7] {
b[i], h = byte(h>>48), h<<8
}
return b
}
// Content length N (must be in [252, 65787]) to long used block M field.
func n2m(n int) (m int) {
return n % 0x10000
}
// Long used block M (must be in [0, 65535]) field to content length N.
func m2n(m int) (n int) {
if m <= maxShort {
m += 0x10000
}
return m
}
func bpack(a []byte) []byte {
if cap(a) > len(a) {
return append([]byte(nil), a...)
}
return a
}

101
vendor/github.com/cznic/lldb/memfiler.go generated vendored
View File

@ -1,101 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// A memory-only implementation of Filer.
package lldb
import (
"fmt"
"io"
"github.com/cznic/internal/file"
)
var _ Filer = &MemFiler{}
// MemFiler is a memory backed Filer. It implements BeginUpdate, EndUpdate and
// Rollback as no-ops. MemFiler is not automatically persistent, but it has
// ReadFrom and WriteTo methods.
type MemFiler struct {
fi file.Interface
nest int
}
// NewMemFiler returns a new MemFiler.
func NewMemFiler() *MemFiler {
fi, err := file.OpenMem("")
if err != nil {
return nil
}
return &MemFiler{fi: fi}
}
// BeginUpdate implements Filer.
func (f *MemFiler) BeginUpdate() error {
f.nest++
return nil
}
// Close implements Filer.
func (f *MemFiler) Close() (err error) {
if f.nest != 0 {
return &ErrPERM{(f.Name() + ":Close")}
}
return f.fi.Close()
}
// EndUpdate implements Filer.
func (f *MemFiler) EndUpdate() (err error) {
if f.nest == 0 {
return &ErrPERM{(f.Name() + ": EndUpdate")}
}
f.nest--
return
}
// Name implements Filer.
func (f *MemFiler) Name() string { return fmt.Sprintf("%p.memfiler", f) }
// PunchHole implements Filer.
func (f *MemFiler) PunchHole(off, size int64) (err error) { return nil }
// ReadAt implements Filer.
func (f *MemFiler) ReadAt(b []byte, off int64) (n int, err error) { return f.fi.ReadAt(b, off) }
// ReadFrom is a helper to populate MemFiler's content from r. 'n' reports the
// number of bytes read from 'r'.
func (f *MemFiler) ReadFrom(r io.Reader) (n int64, err error) { return f.fi.ReadFrom(r) }
// Rollback implements Filer.
func (f *MemFiler) Rollback() (err error) { return nil }
// Size implements Filer.
func (f *MemFiler) Size() (int64, error) {
info, err := f.fi.Stat()
if err != nil {
return 0, err
}
return info.Size(), nil
}
// Sync implements Filer.
func (f *MemFiler) Sync() error { return nil }
// Truncate implements Filer.
func (f *MemFiler) Truncate(size int64) (err error) { return f.fi.Truncate(size) }
// WriteAt implements Filer.
func (f *MemFiler) WriteAt(b []byte, off int64) (n int, err error) { return f.fi.WriteAt(b, off) }
// WriteTo is a helper to copy/persist MemFiler's content to w. If w is also
// an io.WriterAt then WriteTo may attempt to _not_ write any big, for some
// value of big, runs of zeros, i.e. it will attempt to punch holes, where
// possible, in `w` if that happens to be a freshly created or to zero length
// truncated OS file. 'n' reports the number of bytes written to 'w'.
func (f *MemFiler) WriteTo(w io.Writer) (n int64, err error) { return f.fi.WriteTo(w) }

130
vendor/github.com/cznic/lldb/osfiler.go generated vendored
View File

@ -1,130 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package lldb
import (
"io"
"os"
"github.com/cznic/mathutil"
)
var _ Filer = (*OSFiler)(nil)
// OSFile is an os.File like minimal set of methods allowing to construct a
// Filer.
type OSFile interface {
Name() string
Stat() (fi os.FileInfo, err error)
Sync() (err error)
Truncate(size int64) (err error)
io.Closer
io.Reader
io.ReaderAt
io.Seeker
io.Writer
io.WriterAt
}
// OSFiler is like a SimpleFileFiler but based on an OSFile.
type OSFiler struct {
f OSFile
nest int
size int64 // not set if < 0
}
// NewOSFiler returns a Filer from an OSFile. This Filer is like the
// SimpleFileFiler, it does not implement the transaction related methods.
func NewOSFiler(f OSFile) (r *OSFiler) {
return &OSFiler{
f: f,
size: -1,
}
}
// BeginUpdate implements Filer.
func (f *OSFiler) BeginUpdate() (err error) {
f.nest++
return nil
}
// Close implements Filer.
func (f *OSFiler) Close() (err error) {
if f.nest != 0 {
return &ErrPERM{(f.Name() + ":Close")}
}
return f.f.Close()
}
// EndUpdate implements Filer.
func (f *OSFiler) EndUpdate() (err error) {
if f.nest == 0 {
return &ErrPERM{(f.Name() + ":EndUpdate")}
}
f.nest--
return
}
// Name implements Filer.
func (f *OSFiler) Name() string {
return f.f.Name()
}
// PunchHole implements Filer.
func (f *OSFiler) PunchHole(off, size int64) (err error) {
return
}
// ReadAt implements Filer.
func (f *OSFiler) ReadAt(b []byte, off int64) (n int, err error) {
return f.f.ReadAt(b, off)
}
// Rollback implements Filer.
func (f *OSFiler) Rollback() (err error) { return }
// Size implements Filer.
func (f *OSFiler) Size() (n int64, err error) {
if f.size < 0 { // boot
fi, err := f.f.Stat()
if err != nil {
return 0, err
}
f.size = fi.Size()
}
return f.size, nil
}
// Sync implements Filer.
func (f *OSFiler) Sync() (err error) {
return f.f.Sync()
}
// Truncate implements Filer.
func (f *OSFiler) Truncate(size int64) (err error) {
if size < 0 {
return &ErrINVAL{"Truncate size", size}
}
f.size = size
return f.f.Truncate(size)
}
// WriteAt implements Filer.
func (f *OSFiler) WriteAt(b []byte, off int64) (n int, err error) {
if f.size < 0 { // boot
fi, err := os.Stat(f.f.Name())
if err != nil {
return 0, err
}
f.size = fi.Size()
}
f.size = mathutil.MaxInt64(f.size, int64(len(b))+off)
return f.f.WriteAt(b, off)
}

192
vendor/github.com/cznic/lldb/perf-4670.log generated vendored
View File

@ -1,192 +0,0 @@
$ benchcmp -mag -changed log-bench-2016-07-11-1221-e572829f log-bench-2016-07-24-1458-74c3b196
benchmark old ns/op new ns/op delta
BenchmarkBTreePut1-4 18052 666 -96.31%
BenchmarkBTreePut8-4 18069 678 -96.25%
BenchmarkBTreePut16-4 17939 724 -95.96%
BenchmarkBTreePut32-4 18523 1353 -92.70%
BenchmarkAllocatorRndFreeSimpleFileFiler1e1-4 6973 3415 -51.03%
BenchmarkAllocatorRndFreeRollbackFiler0-4 1453839 717344 -50.66%
BenchmarkAllocatorRndFreeSimpleFileFiler0-4 6806 3407 -49.94%
BenchmarkAllocatorRndFreeSimpleFileFiler1e3-4 7942 3997 -49.67%
BenchmarkAllocatorAllocRollbackFiler1e1-4 1279565 651636 -49.07%
BenchmarkAllocatorRndFreeSimpleFileFiler1e2-4 7411 3812 -48.56%
BenchmarkAllocatorRndGetSimpleFileFiler1e3-4 2080 1095 -47.36%
BenchmarkAllocatorRndFreeRollbackFiler1e1-4 1364247 719511 -47.26%
BenchmarkAllocatorRndGetSimpleFileFiler1e2-4 1641 868 -47.11%
BenchmarkBTreeGet32-4 350 652 +86.29%
BenchmarkAllocatorAllocRollbackFiler0-4 1226959 660623 -46.16%
BenchmarkBTreeGet1-4 306 567 +85.29%
BenchmarkAllocatorRndFreeRollbackFiler1e2-4 1372538 761810 -44.50%
BenchmarkBTreeSetSimpleFileFiler1e2-4 43789 24569 -43.89%
BenchmarkAllocatorAllocRollbackFiler1e2-4 1255146 714241 -43.09%
BenchmarkBTreeSetRollbackFiler1e3-4 1974529 1144561 -42.03%
BenchmarkBTreeSetSimpleFileFiler1e3-4 43770 25663 -41.37%
BenchmarkBTreeGet8-4 207 352 +70.05%
BenchmarkBTreeSetSimpleFileFiler0-4 38033 22635 -40.49%
BenchmarkBTreeSetSimpleFileFiler1e1-4 38035 22708 -40.30%
BenchmarkAllocatorRndFreeRollbackFiler1e3-4 1486691 893780 -39.88%
BenchmarkBTreeSetRollbackFiler1e1-4 1470871 892358 -39.33%
BenchmarkBTreeGet16-4 223 363 +62.78%
BenchmarkBTreeSetRollbackFiler1e2-4 1589581 982970 -38.16%
BenchmarkAllocatorAllocRollbackFiler1e3-4 1375172 859304 -37.51%
BenchmarkBTreeSetRollbackFiler0-4 1455317 921016 -36.71%
BenchmarkAllocatorAllocSimpleFileFiler1e1-4 1302 829 -36.33%
BenchmarkAllocatorAllocSimpleFileFiler1e2-4 1364 882 -35.34%
BenchmarkAllocatorAllocSimpleFileFiler0-4 1281 866 -32.40%
BenchmarkAllocatorRndGetRollbackFiler1e3-4 4440 3057 -31.15%
BenchmarkAllocatorRndGetSimpleFileFiler1e1-4 810 565 -30.25%
BenchmarkAllocatorRndGetSimpleFileFiler0-4 802 574 -28.43%
BenchmarkAllocatorAllocSimpleFileFiler1e3-4 1812 1306 -27.92%
BenchmarkAllocatorRndGetACIDFiler1e3-4 4158 3047 -26.72%
BenchmarkBTreeSetMemFiler1e2-4 25093 20683 -17.57%
BenchmarkAllocatorAllocACIDFiler1e1-4 4300146 3666822 -14.73%
BenchmarkBTreeSetMemFiler0-4 22518 19238 -14.57%
BenchmarkAllocatorRndFreeACIDFiler1e1-4 4384437 3766451 -14.09%
BenchmarkBTreeSetMemFiler1e1-4 22682 19498 -14.04%
BenchmarkMemFilerRdSeq-4 1888 2169 +14.88%
BenchmarkBTreeSetMemFiler1e3-4 24512 21377 -12.79%
BenchmarkAllocatorAllocMemFiler1e2-4 584 510 -12.67%
BenchmarkAllocatorRndGetRollbackFiler1e1-4 1959 2212 +12.91%
BenchmarkAllocatorAllocACIDFiler1e2-4 4088780 3626368 -11.31%
BenchmarkAllocatorRndFreeMemFiler1e1-4 2502 2246 -10.23%
BenchmarkAllocatorRndFreeMemFiler0-4 2489 2244 -9.84%
BenchmarkAllocatorRndFreeMemFiler1e2-4 2764 2507 -9.30%
BenchmarkAllocatorAllocMemFiler1e1-4 530 483 -8.87%
BenchmarkBTreeSetACIDFiler0-4 4061680 3705069 -8.78%
BenchmarkAllocatorAllocACIDFiler0-4 4048443 3710996 -8.34%
BenchmarkAllocatorRndFreeMemFiler1e3-4 2923 2699 -7.66%
BenchmarkAllocatorRndGetMemFiler1e2-4 941 872 -7.33%
BenchmarkAllocatorRndGetACIDFiler1e1-4 1993 2142 +7.48%
BenchmarkAllocatorRndGetRollbackFiler0-4 2002 2151 +7.44%
BenchmarkAllocatorRndGetACIDFiler0-4 1985 2129 +7.25%
BenchmarkBTreeSetACIDFiler1e2-4 4119577 3846159 -6.64%
BenchmarkAllocatorRndGetMemFiler1e1-4 598 562 -6.02%
BenchmarkAllocatorRndFreeACIDFiler1e2-4 4038721 3814214 -5.56%
BenchmarkAllocatorRndFreeACIDFiler0-4 3787035 3583356 -5.38%
BenchmarkRollbackFiler-4 4.40 4.65 +5.68%
BenchmarkMemFilerWrRand-4 5448 5755 +5.64%
BenchmarkAllocatorRndGetMemFiler0-4 586 557 -4.95%
BenchmarkAllocatorRndGetMemFiler1e3-4 1156 1100 -4.84%
BenchmarkBTreeSetACIDFiler1e1-4 3994640 3801252 -4.84%
BenchmarkAllocatorAllocMemFiler1e0-4 532 507 -4.70%
BenchmarkAllocatorAllocACIDFiler1e3-4 3809395 3634900 -4.58%
BenchmarkAllocatorAllocMemFiler1e3-4 769 736 -4.29%
BenchmarkBTreeSetACIDFiler1e3-4 4076971 3936825 -3.44%
BenchmarkAllocatorRndFreeACIDFiler1e3-4 3943185 4047580 +2.65%
BenchmarkMemFilerRdRand-4 10183 9980 -1.99%
BenchmarkMemFilerWrSeq-4 1494 1473 -1.41%
BenchmarkAllocatorRndGetRollbackFiler1e2-4 2726 2753 +0.99%
BenchmarkAllocatorRndGetACIDFiler1e2-4 2757 2754 -0.11%
benchmark old allocs new allocs delta
BenchmarkBTreeSetACIDFiler0-4 203 53 -73.89%
BenchmarkBTreeSetACIDFiler1e2-4 241 71 -70.54%
BenchmarkBTreeSetACIDFiler1e3-4 263 78 -70.34%
BenchmarkBTreeSetSimpleFileFiler1e3-4 14 36 +157.14%
BenchmarkBTreeSetACIDFiler1e1-4 135 53 -60.74%
BenchmarkBTreeSetSimpleFileFiler1e2-4 15 36 +140.00%
BenchmarkBTreeSetRollbackFiler0-4 49 22 -55.10%
BenchmarkBTreeSetRollbackFiler1e1-4 49 22 -55.10%
BenchmarkBTreeSetSimpleFileFiler0-4 15 33 +120.00%
BenchmarkBTreeSetSimpleFileFiler1e1-4 15 33 +120.00%
BenchmarkBTreeSetRollbackFiler1e2-4 53 25 -52.83%
BenchmarkBTreeSetRollbackFiler1e3-4 56 27 -51.79%
BenchmarkAllocatorRndFreeACIDFiler0-4 63 35 -44.44%
BenchmarkAllocatorRndFreeACIDFiler1e1-4 62 36 -41.94%
BenchmarkAllocatorAllocACIDFiler1e3-4 45 30 -33.33%
BenchmarkAllocatorAllocRollbackFiler0-4 4 6 +50.00%
BenchmarkAllocatorAllocRollbackFiler1e1-4 4 6 +50.00%
BenchmarkAllocatorAllocRollbackFiler1e2-4 4 6 +50.00%
BenchmarkAllocatorRndGetACIDFiler0-4 4 6 +50.00%
BenchmarkAllocatorRndGetACIDFiler1e1-4 4 6 +50.00%
BenchmarkAllocatorRndGetACIDFiler1e2-4 4 6 +50.00%
BenchmarkAllocatorRndGetRollbackFiler0-4 4 6 +50.00%
BenchmarkAllocatorRndGetRollbackFiler1e1-4 4 6 +50.00%
BenchmarkAllocatorRndGetRollbackFiler1e2-4 4 6 +50.00%
BenchmarkBTreeGet1-4 4 6 +50.00%
BenchmarkBTreeGet32-4 5 7 +40.00%
BenchmarkBTreeGet16-4 3 4 +33.33%
BenchmarkBTreeGet8-4 3 4 +33.33%
BenchmarkBTreePut32-4 11 14 +27.27%
BenchmarkAllocatorAllocRollbackFiler1e3-4 5 6 +20.00%
BenchmarkAllocatorRndGetACIDFiler1e3-4 5 6 +20.00%
BenchmarkAllocatorRndGetRollbackFiler1e3-4 5 6 +20.00%
BenchmarkAllocatorRndFreeACIDFiler1e2-4 67 56 -16.42%
BenchmarkAllocatorRndFreeRollbackFiler0-4 7 8 +14.29%
BenchmarkAllocatorRndFreeRollbackFiler1e1-4 7 8 +14.29%
BenchmarkBTreePut1-4 7 8 +14.29%
BenchmarkBTreePut16-4 7 8 +14.29%
BenchmarkBTreePut8-4 7 8 +14.29%
BenchmarkAllocatorRndFreeRollbackFiler1e2-4 8 9 +12.50%
BenchmarkAllocatorRndFreeACIDFiler1e3-4 71 66 -7.04%
BenchmarkAllocatorAllocACIDFiler1e2-4 29 28 -3.45%
BenchmarkAllocatorAllocSimpleFileFiler0-4 0 2 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e1-4 0 2 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e2-4 0 2 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e3-4 0 2 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler0-4 0 7 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e1-4 0 7 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e2-4 0 7 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e3-4 0 7 +Inf%
benchmark old bytes new bytes delta
BenchmarkBTreePut1-4 139367 145 -99.90%
BenchmarkBTreePut8-4 139367 145 -99.90%
BenchmarkBTreePut16-4 139399 177 -99.87%
BenchmarkBTreePut32-4 139592 402 -99.71%
BenchmarkBTreeSetMemFiler1e2-4 19275 319 -98.35%
BenchmarkBTreeSetMemFiler0-4 12928 256 -98.02%
BenchmarkBTreeSetMemFiler1e1-4 12928 259 -98.00%
BenchmarkBTreeSetSimpleFileFiler1e3-4 168 2836 +1588.10%
BenchmarkBTreeSetSimpleFileFiler1e2-4 170 2840 +1570.59%
BenchmarkBTreeSetRollbackFiler0-4 24242 1566 -93.54%
BenchmarkBTreeSetRollbackFiler1e1-4 24243 1567 -93.54%
BenchmarkBTreeSetMemFiler1e3-4 18734 1282 -93.16%
BenchmarkBTreeSetSimpleFileFiler0-4 169 2436 +1341.42%
BenchmarkBTreeSetSimpleFileFiler1e1-4 169 2433 +1339.64%
BenchmarkBTreeSetRollbackFiler1e2-4 26831 1929 -92.81%
BenchmarkBTreeSetRollbackFiler1e3-4 28710 2161 -92.47%
BenchmarkAllocatorRndFreeRollbackFiler1e3-4 4781 897 -81.24%
BenchmarkAllocatorRndFreeRollbackFiler1e2-4 3861 800 -79.28%
BenchmarkAllocatorRndFreeRollbackFiler1e1-4 3421 725 -78.81%
BenchmarkAllocatorRndFreeRollbackFiler0-4 3308 725 -78.08%
BenchmarkRollbackFiler-4 2 9 +350.00%
BenchmarkAllocatorRndGetACIDFiler1e3-4 2123 545 -74.33%
BenchmarkAllocatorRndGetRollbackFiler1e3-4 2123 546 -74.28%
BenchmarkAllocatorAllocRollbackFiler1e3-4 2148 565 -73.70%
BenchmarkAllocatorRndFreeACIDFiler1e3-4 10040 25877 +157.74%
BenchmarkAllocatorRndFreeACIDFiler1e2-4 9123 20659 +126.45%
BenchmarkAllocatorRndGetACIDFiler1e2-4 1000 529 -47.10%
BenchmarkAllocatorRndGetRollbackFiler1e2-4 1000 530 -47.00%
BenchmarkBTreeSetACIDFiler0-4 31790 17014 -46.48%
BenchmarkAllocatorAllocRollbackFiler1e2-4 1005 540 -46.27%
BenchmarkAllocatorRndGetACIDFiler0-4 880 528 -40.00%
BenchmarkAllocatorRndGetACIDFiler1e1-4 880 528 -40.00%
BenchmarkAllocatorRndGetRollbackFiler0-4 880 528 -40.00%
BenchmarkAllocatorRndGetRollbackFiler1e1-4 880 528 -40.00%
BenchmarkAllocatorAllocRollbackFiler1e1-4 889 536 -39.71%
BenchmarkAllocatorAllocRollbackFiler0-4 884 535 -39.48%
BenchmarkBTreeSetACIDFiler1e2-4 34393 27331 -20.53%
BenchmarkBTreeSetACIDFiler1e1-4 21319 17022 -20.16%
BenchmarkAllocatorRndFreeACIDFiler1e1-4 8030 9898 +23.26%
BenchmarkBTreeSetACIDFiler1e3-4 38350 31566 -17.69%
BenchmarkAllocatorAllocACIDFiler1e3-4 8229 6912 -16.00%
BenchmarkAllocatorRndFreeACIDFiler0-4 8330 9867 +18.45%
BenchmarkBTreeGet8-4 128 112 -12.50%
BenchmarkBTreeGet16-4 136 120 -11.76%
BenchmarkBTreeGet1-4 136 152 +11.76%
BenchmarkAllocatorAllocMemFiler1e0-4 16 15 -6.25%
BenchmarkAllocatorAllocACIDFiler0-4 5618 5693 +1.33%
BenchmarkAllocatorAllocACIDFiler1e1-4 5632 5696 +1.14%
BenchmarkAllocatorAllocACIDFiler1e2-4 5880 5824 -0.95%
BenchmarkAllocatorAllocMemFiler1e2-4 112 111 -0.89%
BenchmarkAllocatorAllocMemFiler1e3-4 1009 1008 -0.10%
BenchmarkMemFilerWrRand-4 10750 10743 -0.07%
BenchmarkAllocatorAllocSimpleFileFiler0-4 0 240 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e1-4 0 240 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e2-4 0 240 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e3-4 0 240 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler0-4 0 920 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e1-4 0 920 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e2-4 0 920 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e3-4 0 920 +Inf%

196
vendor/github.com/cznic/lldb/perf-r550.log generated vendored
View File

@ -1,196 +0,0 @@
$ benchcmp -mag -changed log-bench-2016-07-11-1221-e572829f log-bench-2016-07-24-1458-74c3b196
benchmark old ns/op new ns/op delta
BenchmarkBTreePut1-4 62389 1282 -97.95%
BenchmarkBTreePut8-4 61530 1329 -97.84%
BenchmarkBTreePut16-4 61696 1373 -97.77%
BenchmarkBTreePut32-4 58387 2625 -95.50%
BenchmarkAllocatorRndFreeSimpleFileFiler0-4 15873 6442 -59.42%
BenchmarkAllocatorRndFreeSimpleFileFiler1e1-4 15909 6607 -58.47%
BenchmarkAllocatorRndFreeSimpleFileFiler1e2-4 16737 7348 -56.10%
BenchmarkAllocatorRndGetSimpleFileFiler1e2-4 3413 1561 -54.26%
BenchmarkMemFilerWrRand-4 66220 129632 +95.76%
BenchmarkAllocatorAllocSimpleFileFiler1e1-4 3269 1719 -47.42%
BenchmarkAllocatorRndGetSimpleFileFiler1e3-4 4132 2243 -45.72%
BenchmarkAllocatorAllocSimpleFileFiler0-4 3271 1810 -44.67%
BenchmarkAllocatorRndFreeSimpleFileFiler1e3-4 17774 9951 -44.01%
BenchmarkAllocatorAllocSimpleFileFiler1e2-4 3416 1918 -43.85%
BenchmarkBTreeGet1-4 614 1082 +76.22%
BenchmarkBTreeGet32-4 727 1255 +72.63%
BenchmarkBTreeSetSimpleFileFiler1e2-4 115600 71576 -38.08%
BenchmarkAllocatorRndGetSimpleFileFiler1e1-4 1688 1048 -37.91%
BenchmarkAllocatorRndGetSimpleFileFiler0-4 1646 1022 -37.91%
BenchmarkBTreeSetSimpleFileFiler1e3-4 118683 73793 -37.82%
BenchmarkBTreeGet8-4 427 676 +58.31%
BenchmarkBTreeGet16-4 441 691 +56.69%
BenchmarkBTreeSetSimpleFileFiler0-4 101189 66921 -33.87%
BenchmarkBTreeSetSimpleFileFiler1e1-4 101753 67350 -33.81%
BenchmarkRollbackFiler-4 9.25 13.2 +42.70%
BenchmarkAllocatorRndFreeACIDFiler1e3-4 154155028 113354887 -26.47%
BenchmarkAllocatorRndGetRollbackFiler1e3-4 10973 8301 -24.35%
BenchmarkAllocatorRndGetACIDFiler1e3-4 10032 7893 -21.32%
BenchmarkBTreeSetMemFiler1e2-4 81878 64427 -21.31%
BenchmarkMemFilerRdRand-4 48540 61167 +26.01%
BenchmarkAllocatorAllocMemFiler1e1-4 1005 800 -20.40%
BenchmarkBTreeSetRollbackFiler1e1-4 53542893 42893017 -19.89%
BenchmarkAllocatorRndFreeMemFiler1e1-4 4560 3697 -18.93%
BenchmarkAllocatorRndFreeMemFiler0-4 4536 3679 -18.89%
BenchmarkBTreeSetMemFiler1e3-4 80012 65171 -18.55%
BenchmarkBTreeSetRollbackFiler1e2-4 55207335 45530525 -17.53%
BenchmarkAllocatorRndFreeMemFiler1e2-4 5046 4202 -16.73%
BenchmarkAllocatorRndFreeMemFiler1e3-4 5346 4472 -16.35%
BenchmarkBTreeSetMemFiler0-4 69735 59058 -15.31%
BenchmarkAllocatorAllocSimpleFileFiler1e3-4 7935 9344 +17.76%
BenchmarkAllocatorAllocMemFiler1e2-4 1110 944 -14.95%
BenchmarkAllocatorAllocACIDFiler1e2-4 109752059 128224956 +16.83%
BenchmarkAllocatorRndFreeACIDFiler1e2-4 124974458 109609716 -12.29%
BenchmarkAllocatorRndFreeRollbackFiler1e3-4 84862180 74626566 -12.06%
BenchmarkBTreeSetMemFiler1e1-4 66802 59168 -11.43%
BenchmarkAllocatorRndGetRollbackFiler0-4 4616 5158 +11.74%
BenchmarkAllocatorRndFreeRollbackFiler1e2-4 57052848 51130276 -10.38%
BenchmarkAllocatorRndGetMemFiler1e3-4 2520 2270 -9.92%
BenchmarkAllocatorRndGetMemFiler0-4 1097 1003 -8.57%
BenchmarkAllocatorAllocMemFiler1e0-4 964 883 -8.40%
BenchmarkAllocatorRndGetRollbackFiler1e2-4 6504 7097 +9.12%
BenchmarkAllocatorRndGetRollbackFiler1e1-4 4761 5188 +8.97%
BenchmarkAllocatorRndGetMemFiler1e2-4 1715 1577 -8.05%
BenchmarkAllocatorRndGetMemFiler1e1-4 1095 1018 -7.03%
BenchmarkAllocatorRndFreeACIDFiler1e1-4 116914490 108760132 -6.97%
BenchmarkAllocatorRndGetACIDFiler1e1-4 4813 5153 +7.06%
BenchmarkAllocatorAllocMemFiler1e3-4 2132 1992 -6.57%
BenchmarkAllocatorRndGetACIDFiler0-4 4728 4998 +5.71%
BenchmarkBTreeSetACIDFiler1e1-4 109770670 104029058 -5.23%
BenchmarkAllocatorRndGetACIDFiler1e2-4 6461 6808 +5.37%
BenchmarkAllocatorAllocRollbackFiler1e2-4 42659658 44885965 +5.22%
BenchmarkAllocatorRndFreeRollbackFiler0-4 46103955 48326779 +4.82%
BenchmarkAllocatorAllocRollbackFiler0-4 44100144 46221286 +4.81%
BenchmarkBTreeSetACIDFiler0-4 107413862 103212528 -3.91%
BenchmarkBTreeSetRollbackFiler0-4 47988451 46221843 -3.68%
BenchmarkAllocatorRndFreeRollbackFiler1e1-4 43687222 45317995 +3.73%
BenchmarkMemFilerRdSeq-4 10945 11280 +3.06%
BenchmarkAllocatorAllocACIDFiler0-4 108243596 105692910 -2.36%
BenchmarkBTreeSetACIDFiler1e2-4 103100307 105549334 +2.38%
BenchmarkBTreeSetRollbackFiler1e3-4 47756563 46862983 -1.87%
BenchmarkAllocatorRndFreeACIDFiler0-4 107960640 110014433 +1.90%
BenchmarkAllocatorAllocACIDFiler1e1-4 109079753 110713125 +1.50%
BenchmarkBTreeSetACIDFiler1e3-4 105766500 107238849 +1.39%
BenchmarkAllocatorAllocRollbackFiler1e3-4 44952236 45424491 +1.05%
BenchmarkMemFilerWrSeq-4 9855 9770 -0.86%
BenchmarkAllocatorAllocACIDFiler1e3-4 114116546 114886042 +0.67%
BenchmarkAllocatorAllocRollbackFiler1e1-4 43817840 44063415 +0.56%
benchmark old allocs new allocs delta
BenchmarkBTreeSetSimpleFileFiler1e2-4 12 33 +175.00%
BenchmarkBTreeSetSimpleFileFiler1e3-4 12 33 +175.00%
BenchmarkBTreeSetSimpleFileFiler0-4 13 30 +130.77%
BenchmarkBTreeSetSimpleFileFiler1e1-4 13 30 +130.77%
BenchmarkAllocatorRndFreeACIDFiler0-4 59 28 -52.54%
BenchmarkAllocatorRndFreeACIDFiler1e1-4 58 28 -51.72%
BenchmarkBTreeSetACIDFiler1e2-4 70 41 -41.43%
BenchmarkBTreeSetACIDFiler1e3-4 83 52 -37.35%
BenchmarkAllocatorRndFreeACIDFiler1e2-4 63 40 -36.51%
BenchmarkAllocatorRndGetACIDFiler0-4 4 6 +50.00%
BenchmarkAllocatorRndGetACIDFiler1e1-4 4 6 +50.00%
BenchmarkAllocatorRndGetACIDFiler1e2-4 4 6 +50.00%
BenchmarkAllocatorRndGetRollbackFiler0-4 4 6 +50.00%
BenchmarkAllocatorRndGetRollbackFiler1e1-4 4 6 +50.00%
BenchmarkAllocatorRndGetRollbackFiler1e2-4 4 6 +50.00%
BenchmarkBTreeGet1-4 4 6 +50.00%
BenchmarkAllocatorAllocRollbackFiler0-4 5 7 +40.00%
BenchmarkAllocatorAllocRollbackFiler1e1-4 5 7 +40.00%
BenchmarkAllocatorAllocRollbackFiler1e2-4 5 7 +40.00%
BenchmarkBTreeGet32-4 5 7 +40.00%
BenchmarkAllocatorAllocACIDFiler1e3-4 49 36 -26.53%
BenchmarkBTreeGet16-4 3 4 +33.33%
BenchmarkBTreeGet8-4 3 4 +33.33%
BenchmarkBTreePut32-4 11 14 +27.27%
BenchmarkAllocatorRndFreeRollbackFiler0-4 5 6 +20.00%
BenchmarkAllocatorRndFreeRollbackFiler1e1-4 5 6 +20.00%
BenchmarkAllocatorRndGetACIDFiler1e3-4 5 6 +20.00%
BenchmarkAllocatorRndGetRollbackFiler1e3-4 5 6 +20.00%
BenchmarkBTreeSetRollbackFiler0-4 10 12 +20.00%
BenchmarkBTreeSetRollbackFiler1e1-4 10 12 +20.00%
BenchmarkBTreeSetRollbackFiler1e2-4 12 14 +16.67%
BenchmarkAllocatorAllocRollbackFiler1e3-4 7 8 +14.29%
BenchmarkBTreePut1-4 7 8 +14.29%
BenchmarkBTreePut16-4 7 8 +14.29%
BenchmarkBTreePut8-4 7 8 +14.29%
BenchmarkAllocatorRndFreeACIDFiler1e3-4 65 58 -10.77%
BenchmarkAllocatorRndFreeRollbackFiler1e3-4 9 10 +11.11%
BenchmarkBTreeSetACIDFiler0-4 35 38 +8.57%
BenchmarkBTreeSetACIDFiler1e1-4 35 38 +8.57%
BenchmarkBTreeSetMemFiler1e3-4 14 13 -7.14%
BenchmarkBTreeSetRollbackFiler1e3-4 16 15 -6.25%
BenchmarkAllocatorAllocACIDFiler1e1-4 32 34 +6.25%
BenchmarkAllocatorAllocACIDFiler0-4 31 32 +3.23%
BenchmarkAllocatorAllocACIDFiler1e2-4 34 33 -2.94%
BenchmarkAllocatorAllocSimpleFileFiler0-4 0 2 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e1-4 0 2 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e2-4 0 2 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e3-4 0 2 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler0-4 0 7 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e1-4 0 7 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e2-4 0 7 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e3-4 0 7 +Inf%
benchmark old bytes new bytes delta
BenchmarkBTreePut1-4 139363 145 -99.90%
BenchmarkBTreePut8-4 139363 146 -99.90%
BenchmarkBTreePut16-4 139397 178 -99.87%
BenchmarkBTreePut32-4 139587 408 -99.71%
BenchmarkBTreeSetMemFiler1e2-4 20202 348 -98.28%
BenchmarkBTreeSetMemFiler0-4 12286 247 -97.99%
BenchmarkBTreeSetMemFiler1e1-4 10617 247 -97.67%
BenchmarkBTreeSetMemFiler1e3-4 20591 1343 -93.48%
BenchmarkBTreeSetSimpleFileFiler1e1-4 164 2299 +1301.83%
BenchmarkBTreeSetSimpleFileFiler0-4 164 2293 +1298.17%
BenchmarkBTreeSetSimpleFileFiler1e2-4 215 2757 +1182.33%
BenchmarkBTreeSetSimpleFileFiler1e3-4 223 2756 +1135.87%
BenchmarkRollbackFiler-4 2 9 +350.00%
BenchmarkAllocatorRndFreeRollbackFiler1e2-4 3273 840 -74.34%
BenchmarkAllocatorRndGetACIDFiler1e3-4 2125 549 -74.16%
BenchmarkAllocatorRndGetRollbackFiler1e3-4 2125 550 -74.12%
BenchmarkAllocatorRndFreeRollbackFiler1e3-4 4182 1248 -70.16%
BenchmarkAllocatorRndFreeRollbackFiler0-4 2026 683 -66.29%
BenchmarkAllocatorRndFreeRollbackFiler1e1-4 1904 683 -64.13%
BenchmarkAllocatorRndFreeACIDFiler1e3-4 9449 21867 +131.42%
BenchmarkMemFilerWrRand-4 76195 168878 +121.64%
BenchmarkAllocatorRndGetSimpleFileFiler1e3-4 2 1 -50.00%
BenchmarkAllocatorRndGetACIDFiler1e2-4 1000 530 -47.00%
BenchmarkAllocatorRndGetRollbackFiler1e2-4 1000 532 -46.80%
BenchmarkBTreeSetACIDFiler1e1-4 21011 36013 +71.40%
BenchmarkBTreeSetACIDFiler0-4 21011 36007 +71.37%
BenchmarkAllocatorRndGetACIDFiler0-4 880 528 -40.00%
BenchmarkAllocatorRndGetACIDFiler1e1-4 880 528 -40.00%
BenchmarkAllocatorRndGetRollbackFiler1e1-4 880 528 -40.00%
BenchmarkAllocatorRndGetRollbackFiler0-4 880 529 -39.89%
BenchmarkBTreeSetRollbackFiler0-4 6690 10904 +62.99%
BenchmarkBTreeSetRollbackFiler1e1-4 6696 10904 +62.84%
BenchmarkBTreeSetACIDFiler1e3-4 27500 44167 +60.61%
BenchmarkBTreeSetACIDFiler1e2-4 22726 36274 +59.61%
BenchmarkAllocatorAllocRollbackFiler1e3-4 3100 1978 -36.19%
BenchmarkAllocatorRndFreeACIDFiler1e2-4 8440 12086 +43.20%
BenchmarkBTreeSetRollbackFiler1e2-4 8116 11223 +38.28%
BenchmarkAllocatorRndFreeACIDFiler1e1-4 7266 5823 -19.86%
BenchmarkAllocatorRndFreeACIDFiler0-4 7168 5769 -19.52%
BenchmarkAllocatorAllocRollbackFiler1e2-4 1382 1122 -18.81%
BenchmarkAllocatorAllocRollbackFiler0-4 1185 1022 -13.76%
BenchmarkAllocatorAllocRollbackFiler1e1-4 1196 1053 -11.96%
BenchmarkBTreeGet8-4 127 112 -11.81%
BenchmarkBTreeGet16-4 136 120 -11.76%
BenchmarkBTreeGet1-4 136 152 +11.76%
BenchmarkBTreeSetRollbackFiler1e3-4 10761 12007 +11.58%
BenchmarkAllocatorAllocACIDFiler1e1-4 6544 7264 +11.00%
BenchmarkAllocatorAllocACIDFiler0-4 6512 7144 +9.71%
BenchmarkAllocatorAllocACIDFiler1e2-4 6980 7459 +6.86%
BenchmarkMemFilerWrSeq-4 50 51 +2.00%
BenchmarkAllocatorAllocACIDFiler1e3-4 11037 10948 -0.81%
BenchmarkAllocatorAllocMemFiler1e3-4 1009 1008 -0.10%
BenchmarkAllocatorAllocSimpleFileFiler0-4 0 240 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e1-4 0 240 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e2-4 0 240 +Inf%
BenchmarkAllocatorAllocSimpleFileFiler1e3-4 0 240 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler0-4 0 920 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e1-4 0 920 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e2-4 0 920 +Inf%
BenchmarkAllocatorRndFreeSimpleFileFiler1e3-4 0 920 +Inf%
BenchmarkMemFilerRdRand-4 0 8 +Inf%

99
vendor/github.com/cznic/lldb/simplefilefiler.go generated vendored
View File

@ -1,99 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// A basic os.File backed Filer.
package lldb
import (
"os"
"github.com/cznic/internal/file"
)
var _ Filer = &SimpleFileFiler{}
// SimpleFileFiler is an os.File backed Filer intended for use where structural
// consistency can be reached by other means (SimpleFileFiler is for example
// wrapped in eg. an RollbackFiler or ACIDFiler0) or where persistence is not
// required (temporary/working data sets).
//
// SimpleFileFiler is the most simple os.File backed Filer implementation as it
// does not really implement BeginUpdate and EndUpdate/Rollback in any way
// which would protect the structural integrity of data. If misused e.g. as a
// real database storage w/o other measures, it can easily cause data loss
// when, for example, a power outage occurs or the updating process terminates
// abruptly.
type SimpleFileFiler struct {
fi file.Interface
name string
nest int
}
// NewSimpleFileFiler returns a new SimpleFileFiler.
func NewSimpleFileFiler(f *os.File) *SimpleFileFiler {
fi, err := file.Open(f)
if err != nil {
return nil
}
sf := &SimpleFileFiler{fi: fi, name: f.Name()}
return sf
}
// BeginUpdate implements Filer.
func (f *SimpleFileFiler) BeginUpdate() error {
f.nest++
return nil
}
// Close implements Filer.
func (f *SimpleFileFiler) Close() (err error) {
if f.nest != 0 {
return &ErrPERM{(f.Name() + ":Close")}
}
return f.fi.Close()
}
// EndUpdate implements Filer.
func (f *SimpleFileFiler) EndUpdate() (err error) {
if f.nest == 0 {
return &ErrPERM{(f.Name() + ":EndUpdate")}
}
f.nest--
return
}
// Name implements Filer.
func (f *SimpleFileFiler) Name() string { return f.name }
// PunchHole implements Filer.
func (f *SimpleFileFiler) PunchHole(off, size int64) (err error) { return nil }
// ReadAt implements Filer.
func (f *SimpleFileFiler) ReadAt(b []byte, off int64) (n int, err error) { return f.fi.ReadAt(b, off) }
// Rollback implements Filer.
func (f *SimpleFileFiler) Rollback() (err error) { return nil }
// Size implements Filer.
func (f *SimpleFileFiler) Size() (int64, error) {
info, err := f.fi.Stat()
if err != nil {
return 0, err
}
return info.Size(), nil
}
// Sync implements Filer.
func (f *SimpleFileFiler) Sync() error { return f.fi.Sync() }
// Truncate implements Filer.
func (f *SimpleFileFiler) Truncate(size int64) (err error) { return f.fi.Truncate(size) }
// WriteAt implements Filer.
func (f *SimpleFileFiler) WriteAt(b []byte, off int64) (n int, err error) { return f.fi.WriteAt(b, off) }

615
vendor/github.com/cznic/lldb/xact.go generated vendored
View File

@ -1,615 +0,0 @@
// Copyright 2014 The lldb Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Structural transactions.
package lldb
//DONE+ TransactionalMemoryFiler
// ----
// Use NewRollbackFiler(myMemFiler, ...)
/*
bfBits: 3
BenchmarkRollbackFiler 20000000 102 ns/op 9.73 MB/s
bfBits: 4
BenchmarkRollbackFiler 50000000 55.7 ns/op 17.95 MB/s
bfBits: 5
BenchmarkRollbackFiler 100000000 32.2 ns/op 31.06 MB/s
bfBits: 6
BenchmarkRollbackFiler 100000000 20.6 ns/op 48.46 MB/s
bfBits: 7
BenchmarkRollbackFiler 100000000 15.1 ns/op 66.12 MB/s
bfBits: 8
BenchmarkRollbackFiler 100000000 10.5 ns/op 95.66 MB/s
bfBits: 9
BenchmarkRollbackFiler 200000000 8.02 ns/op 124.74 MB/s
bfBits: 10
BenchmarkRollbackFiler 200000000 9.25 ns/op 108.09 MB/s
bfBits: 11
BenchmarkRollbackFiler 100000000 11.7 ns/op 85.47 MB/s
bfBits: 12
BenchmarkRollbackFiler 100000000 17.2 ns/op 57.99 MB/s
bfBits: 13
BenchmarkRollbackFiler 100000000 32.7 ns/op 30.58 MB/s
bfBits: 14
BenchmarkRollbackFiler 50000000 39.6 ns/op 25.27 MB/s
*/
import (
"fmt"
"io"
"sync"
"github.com/cznic/fileutil"
"github.com/cznic/internal/buffer"
"github.com/cznic/mathutil"
)
var (
_ Filer = &bitFiler{} // Ensure bitFiler is a Filer.
_ Filer = &RollbackFiler{} // ditto
)
const (
bfBits = 12
bfSize = 1 << bfBits
bfMask = bfSize - 1
)
type (
bitPage struct {
prev, next *bitPage
pdata *[]byte
data []byte
dirty bool
}
bitFilerMap map[int64]*bitPage
bitFiler struct {
parent Filer
m bitFilerMap
size int64
}
)
func newBitFiler(parent Filer) (f *bitFiler, err error) {
sz, err := parent.Size()
if err != nil {
return
}
return &bitFiler{parent: parent, m: bitFilerMap{}, size: sz}, nil
}
func (f *bitFiler) BeginUpdate() error { panic("internal error") }
func (f *bitFiler) EndUpdate() error { panic("internal error") }
func (f *bitFiler) Rollback() error { panic("internal error") }
func (f *bitFiler) Sync() error { panic("internal error") }
func (f *bitFiler) Close() (err error) { return }
func (f *bitFiler) Name() string { return fmt.Sprintf("%p.bitfiler", f) }
func (f *bitFiler) Size() (int64, error) { return f.size, nil }
func (f *bitFiler) free() {
for _, pg := range f.m {
buffer.Put(pg.pdata)
}
}
func (f *bitFiler) PunchHole(off, size int64) (err error) {
first := off >> bfBits
if off&bfMask != 0 {
first++
}
off += size - 1
last := off >> bfBits
if off&bfMask != 0 {
last--
}
if limit := f.size >> bfBits; last > limit {
last = limit
}
for pgI := first; pgI <= last; pgI++ {
pg := &bitPage{}
pg.pdata = buffer.CGet(bfSize)
pg.data = *pg.pdata
pg.dirty = true
f.m[pgI] = pg
}
return
}
func (f *bitFiler) ReadAt(b []byte, off int64) (n int, err error) {
avail := f.size - off
pgI := off >> bfBits
pgO := int(off & bfMask)
rem := len(b)
if int64(rem) >= avail {
rem = int(avail)
err = io.EOF
}
for rem != 0 && avail > 0 {
pg := f.m[pgI]
if pg == nil {
pg = &bitPage{}
pg.pdata = buffer.CGet(bfSize)
pg.data = *pg.pdata
if f.parent != nil {
_, err = f.parent.ReadAt(pg.data, off&^bfMask)
if err != nil && !fileutil.IsEOF(err) {
return
}
err = nil
}
f.m[pgI] = pg
}
nc := copy(b[:mathutil.Min(rem, bfSize)], pg.data[pgO:])
pgI++
pgO = 0
rem -= nc
n += nc
b = b[nc:]
off += int64(nc)
}
return
}
func (f *bitFiler) Truncate(size int64) (err error) {
switch {
case size < 0:
return &ErrINVAL{"Truncate size", size}
case size == 0:
f.m = bitFilerMap{}
f.size = 0
return
}
first := size >> bfBits
if size&bfMask != 0 {
first++
}
last := f.size >> bfBits
if f.size&bfMask != 0 {
last++
}
for ; first < last; first++ {
if bp, ok := f.m[first]; ok {
buffer.Put(bp.pdata)
}
delete(f.m, first)
}
f.size = size
return
}
func (f *bitFiler) WriteAt(b []byte, off int64) (n int, err error) {
off0 := off
pgI := off >> bfBits
pgO := int(off & bfMask)
n = len(b)
rem := n
var nc int
for rem != 0 {
pg := f.m[pgI]
if pg == nil {
pg = &bitPage{}
pg.pdata = buffer.CGet(bfSize)
pg.data = *pg.pdata
if f.parent != nil {
_, err = f.parent.ReadAt(pg.data, off&^bfMask)
if err != nil && !fileutil.IsEOF(err) {
return
}
err = nil
}
f.m[pgI] = pg
}
nc = copy(pg.data[pgO:], b)
pgI++
pg.dirty = true
pgO = 0
rem -= nc
b = b[nc:]
off += int64(nc)
}
f.size = mathutil.MaxInt64(f.size, off0+int64(n))
return
}
func (f *bitFiler) link() {
for pgI, pg := range f.m {
nx, ok := f.m[pgI+1]
if !ok || !nx.dirty {
continue
}
nx.prev, pg.next = pg, nx
}
}
func (f *bitFiler) dumpDirty(w io.WriterAt) (nwr int, err error) {
f.link()
for pgI, pg := range f.m {
if !pg.dirty {
continue
}
for pg.prev != nil && pg.prev.dirty {
pg = pg.prev
pgI--
}
for pg != nil && pg.dirty {
if _, err := w.WriteAt(pg.data, pgI<<bfBits); err != nil {
return 0, err
}
nwr++
pg.dirty = false
pg = pg.next
pgI++
}
}
return
}
// RollbackFiler is a Filer implementing structural transaction handling.
// Structural transactions should be small and short lived because all non
// committed data are held in memory until committed or discarded by a
// Rollback.
//
// While using RollbackFiler, every intended update of the wrapped Filler, by
// WriteAt, Truncate or PunchHole, _must_ be made within a transaction.
// Attempts to do it outside of a transaction will return ErrPERM. OTOH,
// invoking ReadAt outside of a transaction is not a problem.
//
// No nested transactions: All updates within a transaction are held in memory.
// On a matching EndUpdate the updates held in memory are actually written to
// the wrapped Filer.
//
// Nested transactions: Correct data will be seen from RollbackFiler when any
// level of a nested transaction is rollbacked. The actual writing to the
// wrapped Filer happens only when the outer most transaction nesting level is
// closed.
//
// Invoking Rollback is an alternative to EndUpdate. It discards all changes
// made at the current transaction level and returns the "state" (possibly not
// yet persisted) of the Filer to what it was before the corresponding
// BeginUpdate.
//
// During an open transaction, all reads (using ReadAt) are "dirty" reads,
// seeing the uncommitted changes made to the Filer's data.
//
// Lldb databases should be based upon a RollbackFiler.
//
// With a wrapped MemFiler one gets transactional memory. With, for example a
// wrapped disk based SimpleFileFiler it protects against at least some HW
// errors - if Rollback is properly invoked on such failures and/or if there's
// some WAL or 2PC or whatever other safe mechanism based recovery procedure
// used by the client.
//
// The "real" writes to the wrapped Filer (or WAL instead) go through the
// writerAt supplied to NewRollbackFiler.
//
// List of functions/methods which are recommended to be wrapped in a
// BeginUpdate/EndUpdate structural transaction:
//
// Allocator.Alloc
// Allocator.Free
// Allocator.Realloc
//
// CreateBTree
// RemoveBTree
// BTree.Clear
// BTree.Delete
// BTree.DeleteAny
// BTree.Clear
// BTree.Extract
// BTree.Get (it can mutate the DB)
// BTree.Put
// BTree.Set
//
// NOTE: RollbackFiler is a generic solution intended to wrap Filers provided
// by this package which do not implement any of the transactional methods.
// RollbackFiler thus _does not_ invoke any of the transactional methods of its
// wrapped Filer.
//
// RollbackFiler is safe for concurrent use by multiple goroutines.
type RollbackFiler struct {
mu sync.RWMutex
inCallbackMu sync.RWMutex
bitFiler *bitFiler
checkpoint func(int64) error
f Filer
writerAt io.WriterAt
// afterRollback, if not nil, is called after performing Rollback
// without errros.
afterRollback func() error
tlevel int // transaction nesting level, 0 == not in transaction
closed bool
inCallback bool
}
// NewRollbackFiler returns a RollbackFiler wrapping f.
//
// The checkpoint parameter
//
// The checkpoint function is called after closing (by EndUpdate) the upper
// most level open transaction if all calls of writerAt were successful and the
// DB (or eg. a WAL) is thus now in a consistent state (virtually, in the ideal
// world with no write caches, no HW failures, no process crashes, ...).
//
// NOTE: In, for example, a 2PC it is necessary to reflect also the sz
// parameter as the new file size (as in the parameter to Truncate). All
// changes were successfully written already by writerAt before invoking
// checkpoint.
//
// The writerAt parameter
//
// The writerAt interface is used to commit the updates of the wrapped Filer.
// If any invocation of writerAt fails then a non nil error will be returned
// from EndUpdate and checkpoint will _not_ ne called. Neither is necessary to
// call Rollback. The rule of thumb: The [structural] transaction [level] is
// closed by invoking exactly once one of EndUpdate _or_ Rollback.
//
// It is presumed that writerAt uses WAL or 2PC or whatever other safe
// mechanism to physically commit the updates.
//
// Updates performed by invocations of writerAt are byte-precise, but not
// necessarily maximum possible length precise. IOW, for example an update
// crossing page boundaries may be performed by more than one writerAt
// invocation. No offset sorting is performed. This may change if it proves
// to be a problem. Such change would be considered backward compatible.
//
// NOTE: Using RollbackFiler, but failing to ever invoke a matching "closing"
// EndUpdate after an "opening" BeginUpdate means neither writerAt or
// checkpoint will ever get called - with all the possible data loss
// consequences.
func NewRollbackFiler(f Filer, checkpoint func(sz int64) error, writerAt io.WriterAt) (r *RollbackFiler, err error) {
if f == nil || checkpoint == nil || writerAt == nil {
return nil, &ErrINVAL{Src: "lldb.NewRollbackFiler, nil argument"}
}
return &RollbackFiler{
checkpoint: checkpoint,
f: f,
writerAt: writerAt,
}, nil
}
// Implements Filer.
func (r *RollbackFiler) BeginUpdate() (err error) {
r.mu.Lock()
defer r.mu.Unlock()
parent := r.f
if r.tlevel != 0 {
parent = r.bitFiler
}
r.bitFiler, err = newBitFiler(parent)
if err != nil {
return
}
r.tlevel++
return
}
// Implements Filer.
//
// Close will return an error if not invoked at nesting level 0. However, to
// allow emergency closing from eg. a signal handler; if Close is invoked
// within an open transaction(s), it rollbacks any non committed open
// transactions and performs the Close operation.
//
// IOW: Regardless of the transaction nesting level the Close is always
// performed but any uncommitted transaction data are lost.
func (r *RollbackFiler) Close() (err error) {
r.mu.Lock()
defer r.mu.Unlock()
if r.closed {
return &ErrPERM{r.f.Name() + ": Already closed"}
}
r.closed = true
if err = r.f.Close(); err != nil {
return
}
if r.tlevel != 0 {
err = &ErrPERM{r.f.Name() + ": Close inside an open transaction"}
}
if r.bitFiler != nil {
r.bitFiler.free()
r.bitFiler = nil
}
return
}
// Implements Filer.
func (r *RollbackFiler) EndUpdate() (err error) {
r.mu.Lock()
defer r.mu.Unlock()
if r.tlevel == 0 {
return &ErrPERM{r.f.Name() + " : EndUpdate outside of a transaction"}
}
sz, err := r.size() // Cannot call .Size() -> deadlock
if err != nil {
return
}
r.tlevel--
bf := r.bitFiler
parent := bf.parent
w := r.writerAt
if r.tlevel != 0 {
w = parent
}
nwr, err := bf.dumpDirty(w)
if err != nil {
return
}
switch {
case r.tlevel == 0:
defer func() {
r.bitFiler.free()
r.bitFiler = nil
}()
if nwr == 0 {
return
}
return r.checkpoint(sz)
default:
r.bitFiler.free()
r.bitFiler = parent.(*bitFiler)
sz, _ := bf.Size() // bitFiler.Size() never returns err != nil
return parent.Truncate(sz)
}
}
// Implements Filer.
func (r *RollbackFiler) Name() string {
r.mu.RLock()
defer r.mu.RUnlock()
return r.f.Name()
}
// Implements Filer.
func (r *RollbackFiler) PunchHole(off, size int64) error {
r.mu.Lock()
defer r.mu.Unlock()
if r.tlevel == 0 {
return &ErrPERM{r.f.Name() + ": PunchHole outside of a transaction"}
}
if off < 0 {
return &ErrINVAL{r.f.Name() + ": PunchHole off", off}
}
if size < 0 || off+size > r.bitFiler.size {
return &ErrINVAL{r.f.Name() + ": PunchHole size", size}
}
return r.bitFiler.PunchHole(off, size)
}
// Implements Filer.
func (r *RollbackFiler) ReadAt(b []byte, off int64) (n int, err error) {
r.inCallbackMu.RLock()
defer r.inCallbackMu.RUnlock()
if !r.inCallback {
r.mu.RLock()
defer r.mu.RUnlock()
}
if r.tlevel == 0 {
return r.f.ReadAt(b, off)
}
return r.bitFiler.ReadAt(b, off)
}
// Implements Filer.
func (r *RollbackFiler) Rollback() (err error) {
r.mu.Lock()
defer r.mu.Unlock()
if r.tlevel == 0 {
return &ErrPERM{r.f.Name() + ": Rollback outside of a transaction"}
}
if r.tlevel > 1 {
r.bitFiler.free()
r.bitFiler = r.bitFiler.parent.(*bitFiler)
}
r.tlevel--
if f := r.afterRollback; f != nil {
r.inCallbackMu.Lock()
r.inCallback = true
r.inCallbackMu.Unlock()
defer func() {
r.inCallbackMu.Lock()
r.inCallback = false
r.inCallbackMu.Unlock()
}()
return f()
}
return
}
func (r *RollbackFiler) size() (sz int64, err error) {
if r.tlevel == 0 {
return r.f.Size()
}
return r.bitFiler.Size()
}
// Implements Filer.
func (r *RollbackFiler) Size() (sz int64, err error) {
r.mu.Lock()
defer r.mu.Unlock()
return r.size()
}
// Implements Filer.
func (r *RollbackFiler) Sync() error {
r.mu.Lock()
defer r.mu.Unlock()
return r.f.Sync()
}
// Implements Filer.
func (r *RollbackFiler) Truncate(size int64) error {
r.mu.Lock()
defer r.mu.Unlock()
if r.tlevel == 0 {
return &ErrPERM{r.f.Name() + ": Truncate outside of a transaction"}
}
return r.bitFiler.Truncate(size)
}
// Implements Filer.
func (r *RollbackFiler) WriteAt(b []byte, off int64) (n int, err error) {
r.mu.Lock()
defer r.mu.Unlock()
if r.tlevel == 0 {
return 0, &ErrPERM{r.f.Name() + ": WriteAt outside of a transaction"}
}
return r.bitFiler.WriteAt(b, off)
}

12
vendor/github.com/cznic/mathutil/AUTHORS generated vendored
View File

@ -1,12 +0,0 @@
# This file lists authors for copyright purposes. This file is distinct from
# the CONTRIBUTORS files. See the latter for an explanation.
#
# Names should be added to this file as:
# Name or Organization <email address>
#
# The email address is not required for organizations.
#
# Please keep the list sorted.
CZ.NIC z.s.p.o. <kontakt@nic.cz>
Jan Mercl <0xjnml@gmail.com>

10
vendor/github.com/cznic/mathutil/CONTRIBUTORS generated vendored
View File

@ -1,10 +0,0 @@
# This file lists people who contributed code to this repository. The AUTHORS
# file lists the copyright holders; this file lists people.
#
# Names should be added to this file like so:
# Name <email address>
#
# Please keep the list sorted.
Gary Burd <gary@beagledreams.com>
Jan Mercl <0xjnml@gmail.com>

27
vendor/github.com/cznic/mathutil/LICENSE generated vendored
View File

@ -1,27 +0,0 @@
Copyright (c) 2014 The mathutil Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the names of the authors nor the names of the
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

58
vendor/github.com/cznic/mathutil/Makefile generated vendored
View File

@ -1,58 +0,0 @@
# Copyright (c) 2016 The mathutil Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.
.PHONY: all clean cover cpu editor internalError later mem nuke todo edit
grep=--include=*.go --include=*.l --include=*.y --include=*.yy
ngrep='TODOOK\|parser\.go\|scanner\.go\|.*_string\.go'
all: editor
go vet 2>&1 | grep -v $(ngrep) || true
golint 2>&1 | grep -v $(ngrep) || true
make todo
unused . || true
misspell *.go
gosimple || true
codesweep || true
unconvert || true
maligned || true
clean:
go clean
rm -f *~ *.test *.out
cover:
t=$(shell tempfile) ; go test -coverprofile $$t && go tool cover -html $$t && unlink $$t
cpu: clean
go test -run @ -bench . -cpuprofile cpu.out
go tool pprof -lines *.test cpu.out
edit:
@ 1>/dev/null 2>/dev/null gvim -p Makefile *.go
editor:
gofmt -l -s -w *.go
go test
go build
internalError:
egrep -ho '"internal error.*"' *.go | sort | cat -n
later:
@grep -n $(grep) LATER * || true
@grep -n $(grep) MAYBE * || true
mem: clean
go test -run @ -bench . -memprofile mem.out -memprofilerate 1 -timeout 24h
go tool pprof -lines -web -alloc_space *.test mem.out
nuke: clean
go clean -i
todo:
@grep -nr $(grep) ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* * | grep -v $(ngrep) || true
@grep -nr $(grep) TODO * | grep -v $(ngrep) || true
@grep -nr $(grep) BUG * | grep -v $(ngrep) || true
@grep -nr $(grep) [^[:alpha:]]println * | grep -v $(ngrep) || true

10
vendor/github.com/cznic/mathutil/README generated vendored
View File

@ -1,10 +0,0 @@
This is a goinstall-able mirror of modified code already published at:
http://git.nic.cz/redmine/projects/gornd/repository
Packages in this repository:
Install: $ go get github.com/cznic/mathutil
Godocs: http://godoc.org/github.com/cznic/mathutil
Install: $ go get github.com/cznic/mathutil/mersenne
Godocs: http://godoc.org/github.com/cznic/mathutil/mersenne

207
vendor/github.com/cznic/mathutil/bits.go generated vendored
View File

@ -1,207 +0,0 @@
// Copyright (c) 2014 The mathutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package mathutil
import (
"math/big"
)
// BitLenByte returns the bit width of the non zero part of n.
func BitLenByte(n byte) int {
return log2[n] + 1
}
// BitLenUint16 returns the bit width of the non zero part of n.
func BitLenUint16(n uint16) int {
if b := n >> 8; b != 0 {
return log2[b] + 8 + 1
}
return log2[n] + 1
}
// BitLenUint32 returns the bit width of the non zero part of n.
func BitLenUint32(n uint32) int {
if b := n >> 24; b != 0 {
return log2[b] + 24 + 1
}
if b := n >> 16; b != 0 {
return log2[b] + 16 + 1
}
if b := n >> 8; b != 0 {
return log2[b] + 8 + 1
}
return log2[n] + 1
}
// BitLen returns the bit width of the non zero part of n.
func BitLen(n int) int { // Should handle correctly [future] 64 bit Go ints
if IntBits == 64 {
return BitLenUint64(uint64(n))
}
if b := byte(n >> 24); b != 0 {
return log2[b] + 24 + 1
}
if b := byte(n >> 16); b != 0 {
return log2[b] + 16 + 1
}
if b := byte(n >> 8); b != 0 {
return log2[b] + 8 + 1
}
return log2[byte(n)] + 1
}
// BitLenUint returns the bit width of the non zero part of n.
func BitLenUint(n uint) int { // Should handle correctly [future] 64 bit Go uints
if IntBits == 64 {
return BitLenUint64(uint64(n))
}
if b := n >> 24; b != 0 {
return log2[b] + 24 + 1
}
if b := n >> 16; b != 0 {
return log2[b] + 16 + 1
}
if b := n >> 8; b != 0 {
return log2[b] + 8 + 1
}
return log2[n] + 1
}
// BitLenUint64 returns the bit width of the non zero part of n.
func BitLenUint64(n uint64) int {
if b := n >> 56; b != 0 {
return log2[b] + 56 + 1
}
if b := n >> 48; b != 0 {
return log2[b] + 48 + 1
}
if b := n >> 40; b != 0 {
return log2[b] + 40 + 1
}
if b := n >> 32; b != 0 {
return log2[b] + 32 + 1
}
if b := n >> 24; b != 0 {
return log2[b] + 24 + 1
}
if b := n >> 16; b != 0 {
return log2[b] + 16 + 1
}
if b := n >> 8; b != 0 {
return log2[b] + 8 + 1
}
return log2[n] + 1
}
// BitLenUintptr returns the bit width of the non zero part of n.
func BitLenUintptr(n uintptr) int {
if b := n >> 56; b != 0 {
return log2[b] + 56 + 1
}
if b := n >> 48; b != 0 {
return log2[b] + 48 + 1
}
if b := n >> 40; b != 0 {
return log2[b] + 40 + 1
}
if b := n >> 32; b != 0 {
return log2[b] + 32 + 1
}
if b := n >> 24; b != 0 {
return log2[b] + 24 + 1
}
if b := n >> 16; b != 0 {
return log2[b] + 16 + 1
}
if b := n >> 8; b != 0 {
return log2[b] + 8 + 1
}
return log2[n] + 1
}
// PopCountByte returns population count of n (number of bits set in n).
func PopCountByte(n byte) int {
return int(popcnt[n])
}
// PopCountUint16 returns population count of n (number of bits set in n).
func PopCountUint16(n uint16) int {
return int(popcnt[byte(n>>8)]) + int(popcnt[byte(n)])
}
// PopCountUint32 returns population count of n (number of bits set in n).
func PopCountUint32(n uint32) int {
return int(popcnt[byte(n>>24)]) + int(popcnt[byte(n>>16)]) +
int(popcnt[byte(n>>8)]) + int(popcnt[byte(n)])
}
// PopCount returns population count of n (number of bits set in n).
func PopCount(n int) int { // Should handle correctly [future] 64 bit Go ints
if IntBits == 64 {
return PopCountUint64(uint64(n))
}
return PopCountUint32(uint32(n))
}
// PopCountUint returns population count of n (number of bits set in n).
func PopCountUint(n uint) int { // Should handle correctly [future] 64 bit Go uints
if IntBits == 64 {
return PopCountUint64(uint64(n))
}
return PopCountUint32(uint32(n))
}
// PopCountUintptr returns population count of n (number of bits set in n).
func PopCountUintptr(n uintptr) int {
if UintPtrBits == 64 {
return PopCountUint64(uint64(n))
}
return PopCountUint32(uint32(n))
}
// PopCountUint64 returns population count of n (number of bits set in n).
func PopCountUint64(n uint64) int {
return int(popcnt[byte(n>>56)]) + int(popcnt[byte(n>>48)]) +
int(popcnt[byte(n>>40)]) + int(popcnt[byte(n>>32)]) +
int(popcnt[byte(n>>24)]) + int(popcnt[byte(n>>16)]) +
int(popcnt[byte(n>>8)]) + int(popcnt[byte(n)])
}
// PopCountBigInt returns population count of |n| (number of bits set in |n|).
func PopCountBigInt(n *big.Int) (r int) {
for _, v := range n.Bits() {
r += PopCountUintptr(uintptr(v))
}
return
}

46
vendor/github.com/cznic/mathutil/envelope.go generated vendored
View File

@ -1,46 +0,0 @@
// Copyright (c) 2014 The mathutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package mathutil
import (
"math"
)
// Approximation type determines approximation methods used by e.g. Envelope.
type Approximation int
// Specific approximation method tags
const (
_ Approximation = iota
Linear // As named
Sinusoidal // Smooth for all derivations
)
// Envelope is an utility for defining simple curves using a small (usually)
// set of data points. Envelope returns a value defined by x, points and
// approximation. The value of x must be in [0,1) otherwise the result is
// undefined or the function may panic. Points are interpreted as dividing the
// [0,1) interval in len(points)-1 sections, so len(points) must be > 1 or the
// function may panic. According to the left and right points closing/adjacent
// to the section the resulting value is interpolated using the chosen
// approximation method. Unsupported values of approximation are silently
// interpreted as 'Linear'.
func Envelope(x float64, points []float64, approximation Approximation) float64 {
step := 1 / float64(len(points)-1)
fslot := math.Floor(x / step)
mod := x - fslot*step
slot := int(fslot)
l, r := points[slot], points[slot+1]
rmod := mod / step
switch approximation {
case Sinusoidal:
k := (math.Sin(math.Pi*(rmod-0.5)) + 1) / 2
return l + (r-l)*k
case Linear:
fallthrough
default:
return l + (r-l)*rmod
}
}

831
vendor/github.com/cznic/mathutil/mathutil.go generated vendored
View File

@ -1,831 +0,0 @@
// Copyright (c) 2014 The mathutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package mathutil provides utilities supplementing the standard 'math' and
// 'math/rand' packages.
//
// Release history and compatibility issues
//
// 2016-10-10: New functions QuadPolyDiscriminant and QuadPolyFactors.
//
// 2013-12-13: The following functions have been REMOVED
//
// func Uint64ToBigInt(n uint64) *big.Int
// func Uint64FromBigInt(n *big.Int) (uint64, bool)
//
// 2013-05-13: The following functions are now DEPRECATED
//
// func Uint64ToBigInt(n uint64) *big.Int
// func Uint64FromBigInt(n *big.Int) (uint64, bool)
//
// These functions will be REMOVED with Go release 1.1+1.
//
// 2013-01-21: The following functions have been REMOVED
//
// func MaxInt() int
// func MinInt() int
// func MaxUint() uint
// func UintPtrBits() int
//
// They are now replaced by untyped constants
//
// MaxInt
// MinInt
// MaxUint
// UintPtrBits
//
// Additionally one more untyped constant was added
//
// IntBits
//
// This change breaks any existing code depending on the above removed
// functions. They should have not been published in the first place, that was
// unfortunate. Instead, defining such architecture and/or implementation
// specific integer limits and bit widths as untyped constants improves
// performance and allows for static dead code elimination if it depends on
// these values. Thanks to minux for pointing it out in the mail list
// (https://groups.google.com/d/msg/golang-nuts/tlPpLW6aJw8/NT3mpToH-a4J).
//
// 2012-12-12: The following functions will be DEPRECATED with Go release
// 1.0.3+1 and REMOVED with Go release 1.0.3+2, b/c of
// http://code.google.com/p/go/source/detail?r=954a79ee3ea8
//
// func Uint64ToBigInt(n uint64) *big.Int
// func Uint64FromBigInt(n *big.Int) (uint64, bool)
package mathutil
import (
"math"
"math/big"
)
// Architecture and/or implementation specific integer limits and bit widths.
const (
MaxInt = 1<<(IntBits-1) - 1
MinInt = -MaxInt - 1
MaxUint = 1<<IntBits - 1
IntBits = 1 << (^uint(0)>>32&1 + ^uint(0)>>16&1 + ^uint(0)>>8&1 + 3)
UintPtrBits = 1 << (^uintptr(0)>>32&1 + ^uintptr(0)>>16&1 + ^uintptr(0)>>8&1 + 3)
)
var (
_1 = big.NewInt(1)
_2 = big.NewInt(2)
)
// GCDByte returns the greatest common divisor of a and b. Based on:
// http://en.wikipedia.org/wiki/Euclidean_algorithm#Implementations
func GCDByte(a, b byte) byte {
for b != 0 {
a, b = b, a%b
}
return a
}
// GCDUint16 returns the greatest common divisor of a and b.
func GCDUint16(a, b uint16) uint16 {
for b != 0 {
a, b = b, a%b
}
return a
}
// GCDUint32 returns the greatest common divisor of a and b.
func GCDUint32(a, b uint32) uint32 {
for b != 0 {
a, b = b, a%b
}
return a
}
// GCDUint64 returns the greatest common divisor of a and b.
func GCDUint64(a, b uint64) uint64 {
for b != 0 {
a, b = b, a%b
}
return a
}
// ISqrt returns floor(sqrt(n)). Typical run time is few hundreds of ns.
func ISqrt(n uint32) (x uint32) {
if n == 0 {
return
}
if n >= math.MaxUint16*math.MaxUint16 {
return math.MaxUint16
}
var px, nx uint32
for x = n; ; px, x = x, nx {
nx = (x + n/x) / 2
if nx == x || nx == px {
break
}
}
return
}
// SqrtUint64 returns floor(sqrt(n)). Typical run time is about 0.5 µs.
func SqrtUint64(n uint64) (x uint64) {
if n == 0 {
return
}
if n >= math.MaxUint32*math.MaxUint32 {
return math.MaxUint32
}
var px, nx uint64
for x = n; ; px, x = x, nx {
nx = (x + n/x) / 2
if nx == x || nx == px {
break
}
}
return
}
// SqrtBig returns floor(sqrt(n)). It panics on n < 0.
func SqrtBig(n *big.Int) (x *big.Int) {
switch n.Sign() {
case -1:
panic(-1)
case 0:
return big.NewInt(0)
}
var px, nx big.Int
x = big.NewInt(0)
x.SetBit(x, n.BitLen()/2+1, 1)
for {
nx.Rsh(nx.Add(x, nx.Div(n, x)), 1)
if nx.Cmp(x) == 0 || nx.Cmp(&px) == 0 {
break
}
px.Set(x)
x.Set(&nx)
}
return
}
// Log2Byte returns log base 2 of n. It's the same as index of the highest
// bit set in n. For n == 0 -1 is returned.
func Log2Byte(n byte) int {
return log2[n]
}
// Log2Uint16 returns log base 2 of n. It's the same as index of the highest
// bit set in n. For n == 0 -1 is returned.
func Log2Uint16(n uint16) int {
if b := n >> 8; b != 0 {
return log2[b] + 8
}
return log2[n]
}
// Log2Uint32 returns log base 2 of n. It's the same as index of the highest
// bit set in n. For n == 0 -1 is returned.
func Log2Uint32(n uint32) int {
if b := n >> 24; b != 0 {
return log2[b] + 24
}
if b := n >> 16; b != 0 {
return log2[b] + 16
}
if b := n >> 8; b != 0 {
return log2[b] + 8
}
return log2[n]
}
// Log2Uint64 returns log base 2 of n. It's the same as index of the highest
// bit set in n. For n == 0 -1 is returned.
func Log2Uint64(n uint64) int {
if b := n >> 56; b != 0 {
return log2[b] + 56
}
if b := n >> 48; b != 0 {
return log2[b] + 48
}
if b := n >> 40; b != 0 {
return log2[b] + 40
}
if b := n >> 32; b != 0 {
return log2[b] + 32
}
if b := n >> 24; b != 0 {
return log2[b] + 24
}
if b := n >> 16; b != 0 {
return log2[b] + 16
}
if b := n >> 8; b != 0 {
return log2[b] + 8
}
return log2[n]
}
// ModPowByte computes (b^e)%m. It panics for m == 0 || b == e == 0.
//
// See also: http://en.wikipedia.org/wiki/Modular_exponentiation#Right-to-left_binary_method
func ModPowByte(b, e, m byte) byte {
if b == 0 && e == 0 {
panic(0)
}
if m == 1 {
return 0
}
r := uint16(1)
for b, m := uint16(b), uint16(m); e > 0; b, e = b*b%m, e>>1 {
if e&1 == 1 {
r = r * b % m
}
}
return byte(r)
}
// ModPowUint16 computes (b^e)%m. It panics for m == 0 || b == e == 0.
func ModPowUint16(b, e, m uint16) uint16 {
if b == 0 && e == 0 {
panic(0)
}
if m == 1 {
return 0
}
r := uint32(1)
for b, m := uint32(b), uint32(m); e > 0; b, e = b*b%m, e>>1 {
if e&1 == 1 {
r = r * b % m
}
}
return uint16(r)
}
// ModPowUint32 computes (b^e)%m. It panics for m == 0 || b == e == 0.
func ModPowUint32(b, e, m uint32) uint32 {
if b == 0 && e == 0 {
panic(0)
}
if m == 1 {
return 0
}
r := uint64(1)
for b, m := uint64(b), uint64(m); e > 0; b, e = b*b%m, e>>1 {
if e&1 == 1 {
r = r * b % m
}
}
return uint32(r)
}
// ModPowUint64 computes (b^e)%m. It panics for m == 0 || b == e == 0.
func ModPowUint64(b, e, m uint64) (r uint64) {
if b == 0 && e == 0 {
panic(0)
}
if m == 1 {
return 0
}
return modPowBigInt(big.NewInt(0).SetUint64(b), big.NewInt(0).SetUint64(e), big.NewInt(0).SetUint64(m)).Uint64()
}
func modPowBigInt(b, e, m *big.Int) (r *big.Int) {
r = big.NewInt(1)
for i, n := 0, e.BitLen(); i < n; i++ {
if e.Bit(i) != 0 {
r.Mod(r.Mul(r, b), m)
}
b.Mod(b.Mul(b, b), m)
}
return
}
// ModPowBigInt computes (b^e)%m. Returns nil for e < 0. It panics for m == 0 || b == e == 0.
func ModPowBigInt(b, e, m *big.Int) (r *big.Int) {
if b.Sign() == 0 && e.Sign() == 0 {
panic(0)
}
if m.Cmp(_1) == 0 {
return big.NewInt(0)
}
if e.Sign() < 0 {
return
}
return modPowBigInt(big.NewInt(0).Set(b), big.NewInt(0).Set(e), m)
}
var uint64ToBigIntDelta big.Int
func init() {
uint64ToBigIntDelta.SetBit(&uint64ToBigIntDelta, 63, 1)
}
var uintptrBits int
func init() {
x := uint64(math.MaxUint64)
uintptrBits = BitLenUintptr(uintptr(x))
}
// UintptrBits returns the bit width of an uintptr at the executing machine.
func UintptrBits() int {
return uintptrBits
}
// AddUint128_64 returns the uint128 sum of uint64 a and b.
func AddUint128_64(a, b uint64) (hi uint64, lo uint64) {
lo = a + b
if lo < a {
hi = 1
}
return
}
// MulUint128_64 returns the uint128 bit product of uint64 a and b.
func MulUint128_64(a, b uint64) (hi, lo uint64) {
/*
2^(2 W) ahi bhi + 2^W alo bhi + 2^W ahi blo + alo blo
FEDCBA98 76543210 FEDCBA98 76543210
---- alo*blo ----
---- alo*bhi ----
---- ahi*blo ----
---- ahi*bhi ----
*/
const w = 32
const m = 1<<w - 1
ahi, bhi, alo, blo := a>>w, b>>w, a&m, b&m
lo = alo * blo
mid1 := alo * bhi
mid2 := ahi * blo
c1, lo := AddUint128_64(lo, mid1<<w)
c2, lo := AddUint128_64(lo, mid2<<w)
_, hi = AddUint128_64(ahi*bhi, mid1>>w+mid2>>w+c1+c2)
return
}
// PowerizeBigInt returns (e, p) such that e is the smallest number for which p
// == b^e is greater or equal n. For n < 0 or b < 2 (0, nil) is returned.
//
// NOTE: Run time for large values of n (above about 2^1e6 ~= 1e300000) can be
// significant and/or unacceptabe. For any smaller values of n the function
// typically performs in sub second time. For "small" values of n (cca bellow
// 2^1e3 ~= 1e300) the same can be easily below 10 µs.
//
// A special (and trivial) case of b == 2 is handled separately and performs
// much faster.
func PowerizeBigInt(b, n *big.Int) (e uint32, p *big.Int) {
switch {
case b.Cmp(_2) < 0 || n.Sign() < 0:
return
case n.Sign() == 0 || n.Cmp(_1) == 0:
return 0, big.NewInt(1)
case b.Cmp(_2) == 0:
p = big.NewInt(0)
e = uint32(n.BitLen() - 1)
p.SetBit(p, int(e), 1)
if p.Cmp(n) < 0 {
p.Mul(p, _2)
e++
}
return
}
bw := b.BitLen()
nw := n.BitLen()
p = big.NewInt(1)
var bb, r big.Int
for {
switch p.Cmp(n) {
case -1:
x := uint32((nw - p.BitLen()) / bw)
if x == 0 {
x = 1
}
e += x
switch x {
case 1:
p.Mul(p, b)
default:
r.Set(_1)
bb.Set(b)
e := x
for {
if e&1 != 0 {
r.Mul(&r, &bb)
}
if e >>= 1; e == 0 {
break
}
bb.Mul(&bb, &bb)
}
p.Mul(p, &r)
}
case 0, 1:
return
}
}
}
// PowerizeUint32BigInt returns (e, p) such that e is the smallest number for
// which p == b^e is greater or equal n. For n < 0 or b < 2 (0, nil) is
// returned.
//
// More info: see PowerizeBigInt.
func PowerizeUint32BigInt(b uint32, n *big.Int) (e uint32, p *big.Int) {
switch {
case b < 2 || n.Sign() < 0:
return
case n.Sign() == 0 || n.Cmp(_1) == 0:
return 0, big.NewInt(1)
case b == 2:
p = big.NewInt(0)
e = uint32(n.BitLen() - 1)
p.SetBit(p, int(e), 1)
if p.Cmp(n) < 0 {
p.Mul(p, _2)
e++
}
return
}
var bb big.Int
bb.SetInt64(int64(b))
return PowerizeBigInt(&bb, n)
}
/*
ProbablyPrimeUint32 returns true if n is prime or n is a pseudoprime to base a.
It implements the Miller-Rabin primality test for one specific value of 'a' and
k == 1.
Wrt pseudocode shown at
http://en.wikipedia.org/wiki/Miller-Rabin_primality_test#Algorithm_and_running_time
Input: n > 3, an odd integer to be tested for primality;
Input: k, a parameter that determines the accuracy of the test
Output: composite if n is composite, otherwise probably prime
write n − 1 as 2^s·d with d odd by factoring powers of 2 from n − 1
LOOP: repeat k times:
pick a random integer a in the range [2, n − 2]
x ← a^d mod n
if x = 1 or x = n − 1 then do next LOOP
for r = 1 .. s − 1
x ← x^2 mod n
if x = 1 then return composite
if x = n − 1 then do next LOOP
return composite
return probably prime
... this function behaves like passing 1 for 'k' and additionally a
fixed/non-random 'a'. Otherwise it's the same algorithm.
See also: http://mathworld.wolfram.com/Rabin-MillerStrongPseudoprimeTest.html
*/
func ProbablyPrimeUint32(n, a uint32) bool {
d, s := n-1, 0
for ; d&1 == 0; d, s = d>>1, s+1 {
}
x := uint64(ModPowUint32(a, d, n))
if x == 1 || uint32(x) == n-1 {
return true
}
for ; s > 1; s-- {
if x = x * x % uint64(n); x == 1 {
return false
}
if uint32(x) == n-1 {
return true
}
}
return false
}
// ProbablyPrimeUint64_32 returns true if n is prime or n is a pseudoprime to
// base a. It implements the Miller-Rabin primality test for one specific value
// of 'a' and k == 1. See also ProbablyPrimeUint32.
func ProbablyPrimeUint64_32(n uint64, a uint32) bool {
d, s := n-1, 0
for ; d&1 == 0; d, s = d>>1, s+1 {
}
x := ModPowUint64(uint64(a), d, n)
if x == 1 || x == n-1 {
return true
}
bx, bn := big.NewInt(0).SetUint64(x), big.NewInt(0).SetUint64(n)
for ; s > 1; s-- {
if x = bx.Mod(bx.Mul(bx, bx), bn).Uint64(); x == 1 {
return false
}
if x == n-1 {
return true
}
}
return false
}
// ProbablyPrimeBigInt_32 returns true if n is prime or n is a pseudoprime to
// base a. It implements the Miller-Rabin primality test for one specific value
// of 'a' and k == 1. See also ProbablyPrimeUint32.
func ProbablyPrimeBigInt_32(n *big.Int, a uint32) bool {
var d big.Int
d.Set(n)
d.Sub(&d, _1) // d <- n-1
s := 0
for ; d.Bit(s) == 0; s++ {
}
nMinus1 := big.NewInt(0).Set(&d)
d.Rsh(&d, uint(s))
x := ModPowBigInt(big.NewInt(int64(a)), &d, n)
if x.Cmp(_1) == 0 || x.Cmp(nMinus1) == 0 {
return true
}
for ; s > 1; s-- {
if x = x.Mod(x.Mul(x, x), n); x.Cmp(_1) == 0 {
return false
}
if x.Cmp(nMinus1) == 0 {
return true
}
}
return false
}
// ProbablyPrimeBigInt returns true if n is prime or n is a pseudoprime to base
// a. It implements the Miller-Rabin primality test for one specific value of
// 'a' and k == 1. See also ProbablyPrimeUint32.
func ProbablyPrimeBigInt(n, a *big.Int) bool {
var d big.Int
d.Set(n)
d.Sub(&d, _1) // d <- n-1
s := 0
for ; d.Bit(s) == 0; s++ {
}
nMinus1 := big.NewInt(0).Set(&d)
d.Rsh(&d, uint(s))
x := ModPowBigInt(a, &d, n)
if x.Cmp(_1) == 0 || x.Cmp(nMinus1) == 0 {
return true
}
for ; s > 1; s-- {
if x = x.Mod(x.Mul(x, x), n); x.Cmp(_1) == 0 {
return false
}
if x.Cmp(nMinus1) == 0 {
return true
}
}
return false
}
// Max returns the larger of a and b.
func Max(a, b int) int {
if a > b {
return a
}
return b
}
// Min returns the smaller of a and b.
func Min(a, b int) int {
if a < b {
return a
}
return b
}
// UMax returns the larger of a and b.
func UMax(a, b uint) uint {
if a > b {
return a
}
return b
}
// UMin returns the smaller of a and b.
func UMin(a, b uint) uint {
if a < b {
return a
}
return b
}
// MaxByte returns the larger of a and b.
func MaxByte(a, b byte) byte {
if a > b {
return a
}
return b
}
// MinByte returns the smaller of a and b.
func MinByte(a, b byte) byte {
if a < b {
return a
}
return b
}
// MaxInt8 returns the larger of a and b.
func MaxInt8(a, b int8) int8 {
if a > b {
return a
}
return b
}
// MinInt8 returns the smaller of a and b.
func MinInt8(a, b int8) int8 {
if a < b {
return a
}
return b
}
// MaxUint16 returns the larger of a and b.
func MaxUint16(a, b uint16) uint16 {
if a > b {
return a
}
return b
}
// MinUint16 returns the smaller of a and b.
func MinUint16(a, b uint16) uint16 {
if a < b {
return a
}
return b
}
// MaxInt16 returns the larger of a and b.
func MaxInt16(a, b int16) int16 {
if a > b {
return a
}
return b
}
// MinInt16 returns the smaller of a and b.
func MinInt16(a, b int16) int16 {
if a < b {
return a
}
return b
}
// MaxUint32 returns the larger of a and b.
func MaxUint32(a, b uint32) uint32 {
if a > b {
return a
}
return b
}
// MinUint32 returns the smaller of a and b.
func MinUint32(a, b uint32) uint32 {
if a < b {
return a
}
return b
}
// MaxInt32 returns the larger of a and b.
func MaxInt32(a, b int32) int32 {
if a > b {
return a
}
return b
}
// MinInt32 returns the smaller of a and b.
func MinInt32(a, b int32) int32 {
if a < b {
return a
}
return b
}
// MaxUint64 returns the larger of a and b.
func MaxUint64(a, b uint64) uint64 {
if a > b {
return a
}
return b
}
// MinUint64 returns the smaller of a and b.
func MinUint64(a, b uint64) uint64 {
if a < b {
return a
}
return b
}
// MaxInt64 returns the larger of a and b.
func MaxInt64(a, b int64) int64 {
if a > b {
return a
}
return b
}
// MinInt64 returns the smaller of a and b.
func MinInt64(a, b int64) int64 {
if a < b {
return a
}
return b
}
// ToBase produces n in base b. For example
//
// ToBase(2047, 22) -> [1, 5, 4]
//
// 1 * 22^0 1
// 5 * 22^1 110
// 4 * 22^2 1936
// ----
// 2047
//
// ToBase panics for bases < 2.
func ToBase(n *big.Int, b int) []int {
var nn big.Int
nn.Set(n)
if b < 2 {
panic("invalid base")
}
k := 1
switch nn.Sign() {
case -1:
nn.Neg(&nn)
k = -1
case 0:
return []int{0}
}
bb := big.NewInt(int64(b))
var r []int
rem := big.NewInt(0)
for nn.Sign() != 0 {
nn.QuoRem(&nn, bb, rem)
r = append(r, k*int(rem.Int64()))
}
return r
}

267
vendor/github.com/cznic/mathutil/nist-sts-2-1-1-report generated vendored
View File

@ -1,267 +0,0 @@
$ ./example -max 100000000 > rnd.dat
$ ./assess 1000000
G E N E R A T O R S E L E C T I O N
______________________________________
[0] Input File [1] Linear Congruential
[2] Quadratic Congruential I [3] Quadratic Congruential II
[4] Cubic Congruential [5] XOR
[6] Modular Exponentiation [7] Blum-Blum-Shub
[8] Micali-Schnorr [9] G Using SHA-1
Enter Choice: 0
User Prescribed Input File: rnd.dat
S T A T I S T I C A L T E S T S
_________________________________
[01] Frequency [02] Block Frequency
[03] Cumulative Sums [04] Runs
[05] Longest Run of Ones [06] Rank
[07] Discrete Fourier Transform [08] Nonperiodic Template Matchings
[09] Overlapping Template Matchings [10] Universal Statistical
[11] Approximate Entropy [12] Random Excursions
[13] Random Excursions Variant [14] Serial
[15] Linear Complexity
INSTRUCTIONS
Enter 0 if you DO NOT want to apply all of the
statistical tests to each sequence and 1 if you DO.
Enter Choice: 1
P a r a m e t e r A d j u s t m e n t s
-----------------------------------------
[1] Block Frequency Test - block length(M): 128
[2] NonOverlapping Template Test - block length(m): 9
[3] Overlapping Template Test - block length(m): 9
[4] Approximate Entropy Test - block length(m): 10
[5] Serial Test - block length(m): 16
[6] Linear Complexity Test - block length(M): 500
Select Test (0 to continue): 0
How many bitstreams? 200
Input File Format:
[0] ASCII - A sequence of ASCII 0's and 1's
[1] Binary - Each byte in data file contains 8 bits of data
Select input mode: 1
Statistical Testing In Progress.........
Statistical Testing Complete!!!!!!!!!!!!
$ cat experiments/AlgorithmTesting/finalAnalysisReport.txt
------------------------------------------------------------------------------
RESULTS FOR THE UNIFORMITY OF P-VALUES AND THE PROPORTION OF PASSING SEQUENCES
------------------------------------------------------------------------------
generator is <rnd.dat>
------------------------------------------------------------------------------
C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 P-VALUE PROPORTION STATISTICAL TEST
------------------------------------------------------------------------------
28 22 17 19 15 8 24 23 19 25 0.093720 198/200 Frequency
20 18 24 14 18 17 16 28 21 24 0.504219 199/200 BlockFrequency
25 22 17 24 19 21 22 15 16 19 0.825505 197/200 CumulativeSums
27 17 16 22 14 26 14 25 19 20 0.304126 199/200 CumulativeSums
22 19 14 23 22 22 13 28 13 24 0.224821 199/200 Runs
20 24 18 21 15 13 22 23 24 20 0.719747 197/200 LongestRun
22 26 18 22 26 15 17 22 20 12 0.410055 199/200 Rank
25 22 26 22 20 16 20 20 16 13 0.585209 195/200 FFT
22 11 15 26 33 24 21 13 14 21 0.013102 197/200 NonOverlappingTemplate
17 11 16 27 19 24 19 20 28 19 0.219006 200/200 NonOverlappingTemplate
23 27 24 15 21 11 18 27 15 19 0.162606 197/200 NonOverlappingTemplate
21 18 13 20 19 23 20 17 26 23 0.749884 197/200 NonOverlappingTemplate
24 22 24 24 24 21 13 15 17 16 0.494392 196/200 NonOverlappingTemplate
24 16 23 15 23 18 25 16 18 22 0.699313 199/200 NonOverlappingTemplate
19 23 21 16 27 18 17 20 18 21 0.859637 198/200 NonOverlappingTemplate
12 20 16 19 26 14 30 20 24 19 0.141256 198/200 NonOverlappingTemplate
18 21 17 21 20 14 25 19 24 21 0.859637 198/200 NonOverlappingTemplate
24 25 21 18 23 15 23 17 16 18 0.749884 199/200 NonOverlappingTemplate
20 22 22 18 16 22 28 16 14 22 0.574903 198/200 NonOverlappingTemplate
18 23 22 17 24 25 19 16 23 13 0.626709 199/200 NonOverlappingTemplate
17 22 14 19 21 21 18 19 24 25 0.842937 198/200 NonOverlappingTemplate
18 17 26 21 22 15 22 18 21 20 0.883171 197/200 NonOverlappingTemplate
19 25 16 32 15 19 20 18 16 20 0.236810 199/200 NonOverlappingTemplate
19 18 15 21 24 22 18 21 20 22 0.964295 200/200 NonOverlappingTemplate
21 14 17 23 26 19 20 22 20 18 0.834308 196/200 NonOverlappingTemplate
15 21 17 27 26 23 21 17 24 9 0.129620 198/200 NonOverlappingTemplate
25 17 19 19 18 22 21 22 21 16 0.951205 196/200 NonOverlappingTemplate
20 19 24 21 19 24 16 18 17 22 0.946308 197/200 NonOverlappingTemplate
27 16 19 18 23 19 22 17 22 17 0.807412 197/200 NonOverlappingTemplate
14 18 21 23 23 20 14 22 20 25 0.719747 198/200 NonOverlappingTemplate
18 22 19 12 24 25 25 22 18 15 0.474986 198/200 NonOverlappingTemplate
21 18 23 17 19 18 28 19 20 17 0.825505 198/200 NonOverlappingTemplate
20 19 15 16 27 20 26 17 20 20 0.657933 198/200 NonOverlappingTemplate
17 25 21 21 11 19 22 16 27 21 0.401199 198/200 NonOverlappingTemplate
19 16 15 18 24 19 25 25 19 20 0.769527 199/200 NonOverlappingTemplate
18 20 20 26 20 12 24 25 19 16 0.524101 198/200 NonOverlappingTemplate
14 16 18 23 21 21 19 19 28 21 0.668321 197/200 NonOverlappingTemplate
21 20 23 25 21 22 19 17 14 18 0.875539 197/200 NonOverlappingTemplate
14 16 29 22 23 13 20 29 17 17 0.099513 197/200 NonOverlappingTemplate
14 19 27 19 17 23 18 24 20 19 0.709558 199/200 NonOverlappingTemplate
18 15 21 19 27 22 21 23 17 17 0.779188 198/200 NonOverlappingTemplate
13 23 13 22 22 23 22 21 21 20 0.689019 199/200 NonOverlappingTemplate
17 14 26 26 16 21 30 15 21 14 0.096578 199/200 NonOverlappingTemplate
18 21 24 23 21 13 23 23 19 15 0.719747 197/200 NonOverlappingTemplate
19 21 14 32 20 15 16 18 24 21 0.202268 199/200 NonOverlappingTemplate
27 22 20 21 21 14 15 22 14 24 0.474986 196/200 NonOverlappingTemplate
31 12 25 11 21 18 19 16 24 23 0.050305 197/200 NonOverlappingTemplate
17 26 20 22 15 27 22 19 12 20 0.383827 199/200 NonOverlappingTemplate
15 22 14 14 31 15 27 18 23 21 0.078086 194/200 NonOverlappingTemplate
19 19 14 15 24 21 25 21 20 22 0.788728 197/200 NonOverlappingTemplate
20 21 19 22 25 18 13 24 28 10 0.153763 195/200 NonOverlappingTemplate
23 17 21 25 21 20 13 30 14 16 0.196920 196/200 NonOverlappingTemplate
17 31 17 22 16 15 28 23 11 20 0.050305 197/200 NonOverlappingTemplate
15 21 26 27 15 18 19 21 18 20 0.605916 198/200 NonOverlappingTemplate
23 18 15 14 20 21 20 20 20 29 0.554420 200/200 NonOverlappingTemplate
22 19 19 18 19 17 22 21 31 12 0.311542 199/200 NonOverlappingTemplate
16 22 23 21 19 19 18 24 21 17 0.960198 197/200 NonOverlappingTemplate
21 21 17 20 16 23 25 22 18 17 0.917870 200/200 NonOverlappingTemplate
27 17 17 16 21 20 22 18 21 21 0.859637 197/200 NonOverlappingTemplate
18 24 15 27 18 21 18 16 24 19 0.657933 199/200 NonOverlappingTemplate
13 16 21 21 15 25 18 22 29 20 0.326749 198/200 NonOverlappingTemplate
18 17 23 23 15 19 26 30 11 18 0.125927 198/200 NonOverlappingTemplate
30 21 18 22 17 21 15 17 21 18 0.544254 195/200 NonOverlappingTemplate
12 18 19 24 16 24 18 24 28 17 0.311542 199/200 NonOverlappingTemplate
20 15 23 15 18 30 23 18 17 21 0.410055 196/200 NonOverlappingTemplate
15 18 23 16 29 21 22 16 19 21 0.544254 200/200 NonOverlappingTemplate
18 16 27 13 21 22 22 21 16 24 0.534146 199/200 NonOverlappingTemplate
20 25 18 21 16 21 17 28 21 13 0.484646 200/200 NonOverlappingTemplate
23 22 13 22 14 20 26 18 19 23 0.574903 197/200 NonOverlappingTemplate
21 24 25 13 19 22 18 13 24 21 0.504219 199/200 NonOverlappingTemplate
19 13 18 25 22 15 23 28 19 18 0.410055 195/200 NonOverlappingTemplate
20 15 27 22 26 26 14 13 21 16 0.181557 198/200 NonOverlappingTemplate
18 18 19 23 18 20 19 21 24 20 0.991468 200/200 NonOverlappingTemplate
18 23 17 14 20 25 22 22 22 17 0.816537 198/200 NonOverlappingTemplate
26 15 15 11 23 21 21 16 36 16 0.005557 196/200 NonOverlappingTemplate
27 13 21 23 21 16 19 20 16 24 0.544254 198/200 NonOverlappingTemplate
16 15 32 17 20 23 22 19 20 16 0.262249 200/200 NonOverlappingTemplate
26 19 24 13 24 16 18 18 13 29 0.137282 199/200 NonOverlappingTemplate
15 18 14 27 32 21 15 20 19 19 0.112047 198/200 NonOverlappingTemplate
22 23 22 18 20 23 19 22 16 15 0.924076 196/200 NonOverlappingTemplate
18 17 21 22 14 17 22 24 20 25 0.798139 199/200 NonOverlappingTemplate
15 17 19 24 21 23 17 25 23 16 0.739918 196/200 NonOverlappingTemplate
22 11 15 26 32 25 21 13 14 21 0.017305 197/200 NonOverlappingTemplate
22 16 19 23 22 21 21 19 17 20 0.985788 200/200 NonOverlappingTemplate
22 28 18 24 14 20 23 21 20 10 0.230755 198/200 NonOverlappingTemplate
14 13 22 28 14 28 17 22 23 19 0.129620 197/200 NonOverlappingTemplate
22 16 22 20 21 21 16 19 18 25 0.935716 198/200 NonOverlappingTemplate
15 20 23 17 19 22 21 23 18 22 0.951205 200/200 NonOverlappingTemplate
20 24 21 19 17 19 19 24 15 22 0.930026 198/200 NonOverlappingTemplate
18 21 15 21 17 28 24 22 20 14 0.534146 200/200 NonOverlappingTemplate
19 15 19 19 20 20 15 25 23 25 0.779188 198/200 NonOverlappingTemplate
17 24 25 16 15 21 18 19 23 22 0.788728 198/200 NonOverlappingTemplate
15 20 18 25 24 15 21 31 18 13 0.141256 200/200 NonOverlappingTemplate
24 17 19 20 18 21 15 22 24 20 0.924076 196/200 NonOverlappingTemplate
23 18 17 21 17 28 23 21 18 14 0.605916 197/200 NonOverlappingTemplate
21 19 22 23 16 17 20 21 22 19 0.985788 200/200 NonOverlappingTemplate
27 17 21 27 24 15 15 17 15 22 0.304126 199/200 NonOverlappingTemplate
25 28 20 24 13 14 16 22 19 19 0.304126 197/200 NonOverlappingTemplate
27 16 14 24 22 18 24 20 18 17 0.564639 196/200 NonOverlappingTemplate
18 18 24 19 19 19 26 11 27 19 0.375313 195/200 NonOverlappingTemplate
20 15 29 19 26 16 21 11 18 25 0.141256 197/200 NonOverlappingTemplate
19 14 21 25 11 23 22 25 26 14 0.176657 199/200 NonOverlappingTemplate
18 23 20 17 19 18 29 22 26 8 0.102526 199/200 NonOverlappingTemplate
22 17 18 16 18 20 19 19 25 26 0.834308 198/200 NonOverlappingTemplate
25 18 14 16 16 24 18 18 30 21 0.268917 198/200 NonOverlappingTemplate
24 21 23 13 12 22 20 23 20 22 0.554420 196/200 NonOverlappingTemplate
18 21 21 30 22 17 19 14 18 20 0.534146 197/200 NonOverlappingTemplate
25 20 22 21 15 18 17 20 17 25 0.825505 199/200 NonOverlappingTemplate
18 21 22 21 18 20 26 16 20 18 0.941144 197/200 NonOverlappingTemplate
23 18 22 25 12 16 17 19 26 22 0.474986 198/200 NonOverlappingTemplate
22 18 29 23 19 23 17 17 15 17 0.534146 198/200 NonOverlappingTemplate
19 21 17 26 18 15 22 26 15 21 0.626709 197/200 NonOverlappingTemplate
16 20 20 23 18 21 18 18 25 21 0.955835 199/200 NonOverlappingTemplate
23 21 20 21 22 10 15 27 15 26 0.186566 198/200 NonOverlappingTemplate
18 26 20 26 26 18 17 17 20 12 0.358641 198/200 NonOverlappingTemplate
24 20 21 18 24 12 19 27 14 21 0.401199 195/200 NonOverlappingTemplate
16 25 15 21 24 18 18 25 22 16 0.657933 199/200 NonOverlappingTemplate
24 14 17 26 15 17 17 25 21 24 0.428095 200/200 NonOverlappingTemplate
22 24 11 20 22 24 19 18 12 28 0.176657 196/200 NonOverlappingTemplate
27 16 27 18 27 14 13 16 21 21 0.141256 197/200 NonOverlappingTemplate
23 25 20 18 23 17 15 23 19 17 0.834308 196/200 NonOverlappingTemplate
19 21 20 27 16 16 18 25 16 22 0.678686 199/200 NonOverlappingTemplate
25 22 21 19 15 19 22 19 25 13 0.657933 197/200 NonOverlappingTemplate
19 28 21 25 20 12 18 13 29 15 0.073417 198/200 NonOverlappingTemplate
20 24 21 19 21 15 17 24 20 19 0.941144 198/200 NonOverlappingTemplate
18 29 23 17 24 19 17 18 16 19 0.585209 200/200 NonOverlappingTemplate
18 28 18 16 25 21 18 20 14 22 0.544254 198/200 NonOverlappingTemplate
22 19 23 22 22 21 21 26 12 12 0.401199 199/200 NonOverlappingTemplate
22 15 25 16 21 27 14 22 21 17 0.484646 199/200 NonOverlappingTemplate
18 25 20 23 30 17 13 22 18 14 0.213309 200/200 NonOverlappingTemplate
20 23 21 21 23 29 16 13 16 18 0.410055 199/200 NonOverlappingTemplate
21 19 16 22 31 18 20 17 18 18 0.514124 198/200 NonOverlappingTemplate
26 22 12 14 23 17 21 24 21 20 0.455937 197/200 NonOverlappingTemplate
21 17 18 17 14 32 21 26 18 16 0.162606 197/200 NonOverlappingTemplate
22 24 22 23 11 15 17 18 29 19 0.230755 198/200 NonOverlappingTemplate
19 27 20 19 23 15 24 15 21 17 0.657933 198/200 NonOverlappingTemplate
20 25 16 10 24 13 23 21 21 27 0.149495 200/200 NonOverlappingTemplate
19 21 21 27 17 17 19 21 21 17 0.904708 200/200 NonOverlappingTemplate
18 23 15 19 24 21 23 21 13 23 0.719747 198/200 NonOverlappingTemplate
26 16 28 19 19 18 17 17 16 24 0.474986 199/200 NonOverlappingTemplate
24 32 17 18 20 13 18 18 19 21 0.236810 195/200 NonOverlappingTemplate
26 25 18 17 12 19 20 23 21 19 0.585209 196/200 NonOverlappingTemplate
18 26 25 12 18 16 24 19 18 24 0.410055 199/200 NonOverlappingTemplate
27 21 22 27 21 14 18 14 23 13 0.219006 197/200 NonOverlappingTemplate
18 23 24 16 19 21 16 26 20 17 0.798139 199/200 NonOverlappingTemplate
19 30 15 27 14 19 24 11 22 19 0.073417 198/200 NonOverlappingTemplate
20 23 22 20 22 15 22 21 18 17 0.964295 198/200 NonOverlappingTemplate
22 31 16 26 13 19 17 22 24 10 0.037566 197/200 NonOverlappingTemplate
18 24 22 14 23 19 16 18 19 27 0.637119 197/200 NonOverlappingTemplate
19 20 21 22 21 18 19 22 20 18 0.999438 198/200 NonOverlappingTemplate
27 15 21 18 28 18 15 23 18 17 0.375313 195/200 NonOverlappingTemplate
26 23 20 20 23 19 20 23 14 12 0.514124 199/200 NonOverlappingTemplate
18 19 11 15 21 24 20 26 23 23 0.428095 198/200 NonOverlappingTemplate
19 16 21 25 19 21 15 24 24 16 0.749884 197/200 NonOverlappingTemplate
17 26 23 18 20 26 23 14 18 15 0.494392 198/200 NonOverlappingTemplate
15 17 19 24 21 23 17 25 23 16 0.739918 196/200 NonOverlappingTemplate
26 19 20 20 24 22 22 13 14 20 0.605916 198/200 OverlappingTemplate
29 24 17 21 18 13 18 21 17 22 0.446556 196/200 Universal
22 18 22 20 20 21 22 21 18 16 0.992952 198/200 ApproximateEntropy
14 8 13 9 11 13 13 8 7 10 0.719747 106/106 RandomExcursions
13 18 9 7 12 12 9 6 12 8 0.236810 104/106 RandomExcursions
11 15 10 7 11 14 9 6 12 11 0.595549 106/106 RandomExcursions
15 7 12 12 9 11 16 8 10 6 0.350485 106/106 RandomExcursions
10 10 12 16 10 12 10 7 13 6 0.554420 106/106 RandomExcursions
8 7 12 10 11 16 11 13 10 8 0.657933 106/106 RandomExcursions
9 6 12 12 14 9 11 13 10 10 0.816537 104/106 RandomExcursions
10 10 7 12 11 9 10 13 14 10 0.911413 105/106 RandomExcursions
8 8 12 9 10 5 13 12 17 12 0.319084 104/106 RandomExcursionsVariant
5 11 10 11 7 11 10 15 11 15 0.455937 104/106 RandomExcursionsVariant
6 12 11 8 12 12 12 13 13 7 0.699313 104/106 RandomExcursionsVariant
14 10 11 6 12 9 8 12 11 13 0.779188 104/106 RandomExcursionsVariant
12 12 10 7 17 6 6 12 13 11 0.262249 103/106 RandomExcursionsVariant
13 8 14 13 7 6 6 13 15 11 0.249284 102/106 RandomExcursionsVariant
12 12 12 13 7 9 6 13 12 10 0.739918 105/106 RandomExcursionsVariant
13 15 12 8 9 10 6 9 14 10 0.574903 106/106 RandomExcursionsVariant
10 15 9 12 14 10 8 11 7 10 0.739918 105/106 RandomExcursionsVariant
13 12 8 11 12 11 9 10 11 9 0.978072 103/106 RandomExcursionsVariant
10 13 12 12 8 13 8 9 14 7 0.739918 104/106 RandomExcursionsVariant
12 10 10 14 7 8 7 13 14 11 0.657933 106/106 RandomExcursionsVariant
10 13 10 10 13 10 12 6 10 12 0.897763 106/106 RandomExcursionsVariant
9 12 15 8 13 8 12 8 11 10 0.779188 106/106 RandomExcursionsVariant
9 13 15 10 10 10 8 14 6 11 0.616305 106/106 RandomExcursionsVariant
7 17 9 12 9 11 10 16 4 11 0.129620 106/106 RandomExcursionsVariant
10 9 10 15 7 12 7 8 12 16 0.419021 106/106 RandomExcursionsVariant
9 12 11 8 8 9 15 12 9 13 0.798139 106/106 RandomExcursionsVariant
17 34 11 22 22 17 19 20 13 25 0.026057 199/200 Serial
22 20 16 22 20 18 20 18 23 21 0.989786 199/200 Serial
12 33 25 29 21 11 21 15 14 19 0.003996 199/200 LinearComplexity
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The minimum pass rate for each statistical test with the exception of the
random excursion (variant) test is approximately = 193 for a
sample size = 200 binary sequences.
The minimum pass rate for the random excursion (variant) test
is approximately = 101 for a sample size = 106 binary sequences.
For further guidelines construct a probability table using the MAPLE program
provided in the addendum section of the documentation.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
$

Some files were not shown because too many files have changed in this diff Show More