yyhh.org

Datalevin 1.1.0: State-of-the-Art Performance Across Data Models

2026-09-07T17:00:00+00:00

When we released Datalevin 1.0.0, the message was that one database could handle application state across relational, graph, document, and logical workloads. With Datalevin 1.1.0, we can make a stronger case: that breadth comes with state-of-the-art performance on demanding benchmarks.

The results cover durable transactions, complex relational joins, social graph queries, nested documents, and recursive rules. Datalevin competes with SQLite, PostgreSQL, Neo4j, MongoDB, and dedicated logic engines, using the same database and Datalog query interface.

Here are the headline observations from the benchmark artifacts in the Datalevin repository:

Workload	Datalevin result	Comparison in the measured configuration
Durable transactions	114,739 records/s, synchronous strict WAL, batches of 1,000	3.57× SQLite's throughput
Relational queries	All 113 JOB queries in 38.073 seconds	3.37× as fast as PostgreSQL by total query time
Graph queries	Lower latency on 20 of 21 LDBC-derived read queries	8.56× as fast as Neo4j by summed time; 5.55× by geometric mean
Document reads	11,454 operations/s, one worker, workload C with document queries	3.99× MongoDB, the next fastest system
Logical queries	Lowest latency on all ten selected OpenRuleBench-derived tasks	1.07×–18.11× as fast as the fastest alternative for each task

Each comparison applies to the configuration and workload measured. Together, they make the case for Datalevin as a serious performance choice across data models. The details also show where other systems remain ahead.

Durable transactions

A useful database has to accept changes quickly while maintaining its indexes and honoring its commit promises.

The write benchmark inserts one million person records. Each contains a UUID string identity, first name, last name, and age. SQLite maintains its primary-key index and three explicit value indexes; Datalevin maintains its entity-attribute-value and attribute-value-entity indexes automatically. The comparison aligns logical records and API transaction boundaries, although the physical index work differs.

The chart uses Datalevin's strict WAL profile and SQLite's WAL with synchronous=FULL. Both use their ordinary OS sync behavior; the benchmark's separate macOS fullfsync condition is outside this chart. Data generation, transaction processing, and commit completion contribute to throughput.

For synchronous writes, SQLite is slightly faster at one record per transaction. Datalevin pulls ahead as batches grow: 1.51× at ten records, 1.96× at 100, and 3.57× at 1,000.

The asynchronous API adds another useful capability. It combines queued requests into physical transactions, reaching 245,485 records per second at a request batch size of 100 while retaining strict WAL acknowledgment. This is a different submission pattern from the blocking APIs: multiple requests remain outstanding, so request throughput should not be read as the number of physical commits or as single-request latency.

Concurrent callers also benefit. With four synchronous callers and 1,000-record batches, Datalevin reaches 151,622 records/s, compared with SQLite's 50,735, a 2.99× throughput advantage. In the mixed workload, each iteration looks up a person and upserts a complete record. The blocking strict-WAL paths deliver 5,616 pairs/s for Datalevin and 5,041 for SQLite. Datalevin's asynchronous path reaches 16,996 pairs/s, with reads using the latest available snapshot and no read-your-write barrier between outstanding requests. These results come from the retained concurrent and mixed artifacts.

The practical result is that Datalevin offers both a competitive blocking transaction path and substantial throughput when an application can batch or pipeline its work.

Relational queries: complex joins in 38 seconds

The Join Order Benchmark asks a harder question than point-lookup benchmarks do: can an optimizer choose good plans for complex joins over real, correlated data?

JOB contains 113 queries over an IMDB dataset comprising 21 tables. In Datalevin, the data becomes 277,878,411 datoms. Each engine executes a complete warmup pass followed by a complete measurement pass. PostgreSQL and Datalevin report planning and execution time inside the database, excluding client startup and communication overhead.

Datalevin completes the suite in 38.073 seconds, versus 128.231 seconds for PostgreSQL. SQLite spends 281.849 seconds on the 104 queries it finishes, and nine more queries hit the 60-second cutoff. Charging only that cutoff for each timeout gives SQLite a lower bound of 821.849 seconds, or 21.59× Datalevin's total. The hatched part of its bar makes those timeouts visible.

Datalevin's advantage comes from the suite as a whole: it is faster than PostgreSQL on 65 of 113 queries. PostgreSQL wins the other 48, including some queries where Datalevin's planning overhead is substantial. Datalevin spends 6.689 seconds planning, about 17.6% of its total. There is still room to make planning cheaper while preserving the execution savings.

For applications with complex relationships, the result challenges the idea that moving away from a relational storage model requires giving up relational query performance.

Graph queries: a general database takes on Neo4j

The graph harness implements all 14 Interactive Complex reads and seven Interactive Short reads from LDBC Social Network Benchmark Interactive v1. The SF1 dataset represents a social network; the Neo4j import contains approximately 3.65 million nodes and 20.63 million relationships.

Both engines run embedded, eliminating network transport from this comparison. Each gets a complete warmup pass, followed by a measurement pass in a fresh JVM. Filesystem pages can remain warm, while query parsing and planning are included in the measured call. Final-result caching is disabled.

In the September 1 comparison, Datalevin takes 4.480 seconds across all 21 reads, versus 38.346 seconds for Neo4j Community Embedded 2026.06.0. Datalevin wins 20 queries; Neo4j is about 5% faster on IC10.

The 8.56× summed-time advantage is influenced heavily by IC14. Giving each query equal weight through the geometric mean of its latency ratio still favors Datalevin by 5.55×. Across the seven short reads, the summed-time advantage is 2.95×.

Indexing policy matters here. Neo4j has ID uniqueness constraints and its automatic token lookup indexes, with no workload-specific secondary indexes. Datalevin automatically indexes attribute values. IC6 illustrates the difference: its selected parameter produces an empty result, and Datalevin can use an indexed tag-name lookup. Its 161.67× ratio describes that particular case; it should not be generalized to every graph traversal.

This is an LDBC-derived read-latency study, not an official audited LDBC throughput result. It retains one observation for one bundled parameter per query, and the first query can include lazy compiler initialization in the fresh JVM. All result counts agree; 17 queries also have identical canonical result digests across engines. Four have documented output-representation differences. Those details define what this strong result establishes.

Documents: indexed paths and fast application operations

Datalevin's indexed document type stores nested documents and indexes their paths. An application can query fields, numeric ranges, wildcard paths, and array contents while keeping documents intact.

The document benchmark compares this feature with PostgreSQL JSONB, SQLite JSON1, and MongoDB. It uses 10,000 documents and 10,000 measured operations per pass. The base mixes are reads and updates for A, reads for C, and read-modify-write for F. Each adds document queries with weight 30, producing roughly 23% document queries in the actual schedules.

All systems use explicit durable acknowledgment settings: Datalevin strict WAL, PostgreSQL synchronous_commit=on, SQLite WAL synchronous=FULL, and MongoDB {w: 1, j: true}. PostgreSQL, SQLite, and MongoDB receive indexes for the query mix where supported. Measurements include client-observed execution, transfer, and complete result-ID realization; Datalevin and SQLite are embedded, while PostgreSQL and MongoDB use local servers.

With one worker, Datalevin leads all three mixes. Its 11,454 operations/s on C is 3.99× MongoDB's 2,868, the best alternative. On A and F, its advantage over runner-up PostgreSQL is approximately 1.48× and 1.50×.

The four-worker results show a more varied picture. Datalevin reaches 32,192 operations/s on C, 2.43× PostgreSQL's throughput. PostgreSQL leads A by about 10.5% and F by about 5.3%. Datalevin's strongest advantage here is document querying; concurrent mutation remains an area for further improvement.

The latency breakdown shows where the query advantage comes from.

Datalevin has the lowest p50 for all five query shapes. Nested equality takes 0.057 ms; an any-depth wildcard takes 0.147 ms; array matching takes 0.207 ms. SQLite is competitive on indexed scalar paths, but the two nested-array shapes require scanning documents in this implementation.

Automatic path indexing makes a concrete difference for applications that store evolving, nested records and later need to ask precise questions about their contents.

Logical workloads: recursion and derived relations

Recursive rules are central to Datalog. They express reachability, dependency analysis, and relationships derived from other relationships in a compact form. Their execution can also generate enormous intermediate results.

The portable OpenRuleBench-derived suite tests transitive closure (TC), same generation (SG), and trees of joins (Join1). It compares Datalevin with SQLite, PostgreSQL, XSB, Soufflé, Clara Rules, and O'Doyle Rules under a query-and-full-result-materialization timing boundary. Data loading and program compilation are outside that interval.

The chart uses the September 1 Datalevin 1.1.0 rerun and the August 26 alternative-engine measurements. Input digests and completed answer counts match across those artifacts.

Datalevin has the lowest measured latency in all ten selected tasks. For cyclic transitive closure over 50,000 input facts, it materializes one million result rows in 102.59 ms. The fastest alternative, Soufflé, takes 1,030.36 ms, a 10.04× ratio. For Join1 b1 with both arguments free, Datalevin takes 99.14 ms, versus XSB's 1,795 ms, an 18.11× ratio.

Other leads are much smaller. Join1 b2 is 162.41 ms in Datalevin and 174.00 ms in XSB. The observed 1.07× ratio is useful to report alongside the large wins, especially with only one retained measurement per task.

The suite uses deterministic generated relations following the paper's task definitions; it does not recreate the lost historical input files. These ten tasks are a representative subset, excluding the designated Join1 a free/free stress case and the full scale/binding matrix. Clara's out-of-memory cell and O'Doyle's 60-second timeouts occurred during warmup. Unsupported cells remain marked N/A. Each Clojure wrapper uses an 8 GiB maximum heap; external engines have their own resource configuration.

This is particularly encouraging for a persistent database: expressive rules can deliver performance competitive with specialized logic systems.

What changed in 1.1.0

The release changelog describes improvements throughout the engine: better join-cost estimates, selective indexed lookups, parallel scans, execution in smaller work units, specialized transitive-closure evaluation, and faster batched writes and local identity upserts. Together, these changes target wasted intermediate work, allocation, and transaction overhead.

The release also makes strict durability the default when enabling WAL without an explicit profile. Python and JavaScript gain idiomatic, composable query and transaction APIs. Performance and usability move forward together.

These cross-system results measure the builds recorded in the artifacts. They are not a controlled 1.0-versus-1.1 experiment, so they do not assign a numerical speedup to an individual optimization.

The larger lesson is architectural. Relational joins, graph edges, document paths, and logical rules all benefit when the database can find relevant facts quickly and avoid producing unnecessary intermediate results. Datalevin's fact-based model gives those capabilities a common foundation.

Read the numbers, then try your workload

The measurements were collected on a 12-core Apple Silicon macOS host with Java 21.0.11; the JOB and logic studies identify the machine as an M3 Pro MacBook Pro with 36 GB of memory. They are project-run benchmarks with specific datasets, configurations, and timing boundaries.

The write study includes database growth in one measurement pass with no discarded warmup. JOB, graph, and document studies retain a measurement pass after a separate-process warmup; document runs also warm the newly built database within each pass. Logic uses a complete warmup and measurement in the same child JVM. These protocols produce observations, not confidence intervals, and their different metrics should not be combined into one overall score.

For reproducibility, the charts have a downloadable data snapshot with source-file hashes. The repository links above contain the harnesses and retained artifacts. The graph and logic charts use newer 1.1.0 artifacts than the older tables still present in their benchmark READMEs.

Datalevin 1.1.0 makes a strong case that one database can combine broad expressiveness with leading performance across demanding workloads. That opens up a useful design choice: keep application facts together, and use relations, graphs, documents, and logic wherever each is most natural.

Get Datalevin 1.1.0, explore the guide, and run the benchmark closest to your application. I would love to see what you build with it.

AI Coding Has Erased the Supposed Advantage of Statically Typed Languages

2026-08-13T12:00:00+00:00

For years, advocates of statically typed languages have made the same argument: types catch mistakes earlier, compilers provide better feedback, IDEs offer better assistance, and large codebases become safer to maintain.

That argument rests on an assumption that is rapidly becoming outdated: The person writing the code is human.

AI is not human. It does not prefer Python because Python feels simple. It does not admire Rust because Rust feels rigorous. It has no taste, no emotional attachment, and no programming-language identity.

To an AI, languages differ primarily in how much code, and therefore how many tokens, it must generate to express the same idea.

The more complicated the language, the more tokens it requires. The more tokens it requires, the more opportunities the model has to make a mistake.

It really is that simple.

AI Does Not Need a Type System to Catch Errors

Compiler feedback matters when humans write code.

People forget function signatures. They confuse return types, overlook null values, miss fields, and call methods that do not exist. A type checker acts as a guardrail, catching these mistakes before the program runs.

It is therefore tempting to apply the same logic to AI:

Statically typed languages give AI more feedback, so AI produces better code in them.

This is mostly cargo-cult reasoning inherited from human programming.

When was the last time you saw a capable coding model remain stuck on an ordinary compilation error?

A missing parenthesis, an incorrect primitive type, or a nonexistent method is no longer the central problem in AI-generated software. Such errors occasionally happen, but the model reads the compiler message and fixes them almost immediately.

AI's expensive mistakes are not usually compilation errors. They are misunderstandings.

The model implements the wrong business rule. It overlooks an edge case. It misinterprets the meaning of the data. It breaks an unstated concurrency assumption. It produces a system that is perfectly type-correct and logically wrong. A type checker cannot save you from that.

The claim that AI needs "more compiler feedback" sounds technical, but it often amounts to repeating an old argument without looking at the reality: AI almost always one-shot the code, and compiler feedbacks are not involved for the most part.

Types Are Cost Too

Types are usually described as protection. They are rarely counted as cost.

For AI-generated code, however, a type declaration is first and foremost additional information that must be generated, maintained, and kept consistent.

Types are valuable when they encode real domain constraints:

An order total cannot be negative.
A cancelled transaction cannot be settled again.
An unauthenticated user cannot perform an administrative operation.

But much type information does not express constraints like these. It merely repeats facts that are already obvious from the implementation:

This argument is a string.
This function returns a list of users.
This value might be absent.
This structure implements this interface.

In the human-programming era, this repetition helped programmers understand unfamiliar code. It also allowed IDEs and compilers to catch simple mistakes.

But an AI model is already an extraordinarily capable pattern recognizer. It can often infer these relationships from names, implementations, call sites, tests, and surrounding context.

Requiring the model to state everything again does not automatically improve correctness. It increases output length and adds another consistency obligation.

If a constraint cannot eliminate a meaningful business error but requires dozens of additional tokens, it may be providing ceremony rather than safety.

Compilation Is No Longer the Scarce Capability

The most commonly advertised benefit of static typing is that it moves errors into the compilation stage. In AI-assisted development, compilation errors are among the cheapest errors possible.

The expensive errors are the ones the compiler cannot see:

The requirement was misunderstood.
The tests encode the wrong assumption.
The data model does not reflect the real business.
The API appears reasonable but breaks compatibility.
The concurrent code type-checks but contains a race condition.
The authorization logic compiles but permits unauthorized access.

A program compiling successfully proves only that it satisfies the small subset of rules represented by its type system. Nowadays, this does not buy much, because any frontier AI can almost always meet this narrow requirement in a single shot.

Since AI-generated code already spends very little time stuck on basic compilation failures, continuing to present compiler feedback as a decisive advantage is like advertising a self-driving car on the strength of its gear-change indicator. It may not be entirely useless, but it is simply no longer an important issue.

More Types Means More Tokens

For AI, one of the most meaningful differences between languages is how many tokens are required to express the same behavior.

In a static typed language, a simple operation may require:

Explicit type declarations
Generic parameters
Interfaces or traits
Lifetime annotations
Error-type conversions
Optional-value wrappers
Data-transfer objects
Serialization annotations
Several layers of adapter code

Those additional structures are not free. Longer code requires more generated tokens. More symbols must remain consistent across the context. Changes touch more declarations and more files. Every additional abstraction creates another place where the model can misunderstand the programmer's intent.

AI does not automatically become more correct merely because the code looks more rigorous. It simply now has more things to keep consistent.

More tokens mean more opportunities for error. More abstraction layers mean more room for misunderstanding. More type machinery means more code that does not directly express the business requirement.

On the other hand, dynamic languages may express the same behavior in a lot less number of lines of code. For example, Clojure, a dynamic language, is shown to be the most token efficient in this study.

Language Costs Must Be Recalculated for the AI Era

This does not mean types have no value. Types can document interfaces, define module boundaries, support tooling, and encode genuine domain constraints.

But that cost must now be evaluated honestly. Static typing should not be treated as inherently superior.

Historically, type systems added code and complexity in exchange for reducing human cognitive load and catching human mistakes.

Now, an increasing share of code is generated, modified, and interpreted by AI. AI does not have the same memory limitations, and it rarely remains stuck on syntax or elementary type errors. Its weaknesses lie elsewhere: ambiguous requirements, hidden assumptions, sprawling context, and imperfect semantic understanding.

The old benefit is shrinking while the old cost remains.

And now that tokens are a measurable expense, that cost is more visible than ever.

AI does not care about language ideology. It is not participating in the culture war between static and dynamic typing. It is generating tokens.

If two languages can solve the same problem, but one requires more declarations, more boilerplate, more adapters, and more type gymnastics, that complexity does not disappear. It becomes a longer context, a higher generation cost, and a larger surface area for mistakes.

The supposed advantage of statically typed languages was built on a world in which humans were the primary producers of code. That premise has changed. The conclusion should change with it.

Datalevin 1.0.0 Is Here: One Database for Application State and Agent Memory

2026-07-20T16:00:00+00:00

Today, after six years of development, I am thrilled to announce the availability of Datalevin 1.0.0.

We started Datalevin in 2020 with a deceptively simple question: why should SQL databases remain the default center of application state?

Six years of research, engineering, benchmarking, production use, and community feedback later, Datalevin 1.0.0 is our answer. Datalevin is open source under the Eclipse Public License 2.0. It is a durable, high-performance, fact-first database that brings relational queries, graph traversal, logical reasoning, document access, full-text search, and vector search into one compact system.

The release completes the roadmap we set for 1.0: automatic path indexing for documents; write-ahead logging and transaction-log access; read-only replicas and high availability; a JSON API; broad libraries for Clojure, Java, Python, and JavaScript; and much more.

It also arrives with two new ways to learn Datalevin. The new Datalevin website contains the online guide, with examples in Clojure, Java, Python, and JavaScript. The complete book, Datalevin: The Definitive Guide to Logical and Intelligent Databases, is available in print and ebook formats. In addition to the full database guide, the book contains five chapters devoted to persistent memory for intelligent systems.

This is a release, a book, and a website. More importantly, it is the point at which the original Datalevin idea becomes a complete platform.

Replace SQL at the Center

Datalevin is not intended to be one more specialized database sitting beside a SQL system. Its goal is to replace SQL databases at the center of application state.

That does not mean recreating SQL with different syntax. It means replacing the table as the center of gravity with the fact. Datalevin stores small entity-attribute-value facts, or datoms. The same fact can participate in a row-like record, a graph edge, a nested document workflow, a search result, or a logical rule without being copied into a different data model.

Why make such a fundamental change? There are three main arguments.

1. SQL Is an Awkward Application Interface

SQL is a string-shaped language embedded inside programs. It has a large, English-like syntax, many dialects, and poor composition with host-language code. The enormous ecosystems of ORMs, query builders, migration tools, and object mapping layers are not signs that SQL is a natural application interface. They are evidence of how much machinery is needed to make it behave like one.

Datalevin queries are data. Datalog expresses the facts that must be true, while shared variables create joins implicitly. Rules package reusable logic, and recursive rules use the same form as ordinary queries. The programmer describes relationships instead of spelling out a sequence of join mechanics.

This smaller, more regular surface is easier for people to learn and easier for programs to construct. It is also a better target for AI-generated queries: fewer syntactic branches, fewer vendor-specific choices, and less planner-sensitive ceremony.

2. Table-Shaped Storage Makes Complex Query Planning Harder

Rows bundle many individual facts into containers. When data are sparse, skewed, or correlated, a SQL optimizer has a hard time estimating how many rows will survive each predicate and join. Those cardinality estimates often depend on histograms, independence assumptions, and other approximations. A bad estimate can turn a reasonable query into a huge intermediate result.

A fact-first store begins with explicit, independently indexed data items. Missing facts are absent rather than represented by positional NULL values. Datalevin can count and sample the same indexed facts that query execution will use, giving its cost-based optimizer better raw material for planning complex joins.

This is not only a theoretical advantage. In the Join Order Benchmark, Datalevin has demonstrated that a triplestore can outperform PostgreSQL and SQLite on complex relational queries. The same query engine also performs strongly on recursive logic and industry-standard graph workloads.

3. Stacking Extensions Creates an Integration Tax

Modern SQL databases can add JSON, full-text search, vector indexes, graph features, recursive queries, and procedural extensions. Each capability is useful. The trouble starts when one application question needs several of them at once.

Every extension tends to bring its own syntax, operators, index types, cost model, and operational rules. If the capabilities are split into separate services, the application must also synchronize copies of data and reconcile results across network boundaries. Either way, the glue moves into application code.

Datalevin makes these capabilities composable over one database state in the same elegant fact based model. A single query can ask for documents that contain a phrase, are close to a question in embedding space, belong to a particular entity in a graph, satisfy a nested document predicate, and pass exact permission and lifecycle rules. The database does the integration work where it belongs.

A Memory Substrate for AI Agents

That unified model matters even more for AI agents.

An agent needs more than a transcript and more than a vector database. It needs durable episodes, structured facts, goals, tasks, permissions, tool results, source documents, relationships, and a bounded view of what matters now. It needs to recall by similarity, but it must also know which facts are current, which source supports them, who is allowed to see them, and what they are connected to.

Datalevin is designed to be the memory substrate underneath that system:

Full-text search recalls information by words, phrases, and boolean search expressions.
Vector and embedding search recalls information by semantic similarity.
Logical access uses Datalog queries and rules to enforce exact conditions, derive facts, and reason recursively.
Graph access follows relationships among users, episodes, facts, goals, tasks, evidence, and documents.
Document access keeps nested EDN, JSON, and Markdown values intact while automatically indexing their paths.
Relational access joins structured application state without giving up the fact-first model.

These are not six disconnected products. They are six ways to see and retrieve the same durable fact based state.

That distinction is crucial. Similarity search can find plausible memories, but similarity alone cannot decide whether a fact is authorized, supported, superseded, or relevant to the active goal. Datalevin lets vector and full-text recall produce candidates, then lets logic, graph relationships, document predicates, and ordinary joins constrain and explain the result.

Datalevin does not try to be an agent runtime. The application still owns model calls, tool authorization, ingestion policy, consolidation, truth maintenance, and prompt assembly. Datalevin provides the durable, transactional environment in which those decisions can be stored, inspected, queried, and resumed. Its built-in MCP server can also expose this memory directly to MCP-compatible AI tools.

In other words, a context window is temporary attention. Datalevin is memory.

One Database, Almost the Same API Everywhere

Datalevin began as a Clojure library, but 1.0 is not limited to Clojure applications. The Clojure, Java, Python, and JavaScript APIs now cover almost the same public surface:

Capability	Clojure	Java	Python	JavaScript
Embedded and remote connections	Yes	Yes	Yes	Yes
Datalog query, pull, and explain	Yes	Yes	Yes	Yes
Synchronous and asynchronous transactions	Yes	Yes	Yes	Yes
Datoms, index reads, bulk loading, and re-indexing	Yes	Yes	Yes	Yes
Key-value APIs and explicit KV transactions	Yes	Yes	Yes	Yes
Full-text, vector, embedding, and idoc access	Yes	Yes	Yes	Yes
Standalone search and vector indexes	Yes	Yes	Yes	Yes
UDF registries and query, transaction, and analyzer UDFs	Yes	Yes	Yes	Yes
Backup, snapshots, transaction logs, replicas, and HA administration	Yes	Yes	Yes	Yes

The remaining differences are small and explicit. JavaScript does not expose the Datalog transaction callback because callback re-entry through the Node/JVM bridge can deadlock. Staged mutation of an existing entity object remains a Clojure-only convenience; Java, Python, and JavaScript use transaction maps or builders instead. The full, current list lives in the language compatibility matrix.

Whatever language you choose, the important parts do not change: the same facts, schema, transactions, Datalog queries, and indexes.

Embedded, Server, or Script: Choose at Deployment Time

The data model should not have to change when the deployment topology changes. Datalevin therefore supports three primary ways to run:

Mode	Use it for
Embedded	Link Datalevin into a Clojure, Java, Python, or Node.js process for fast local access, much like SQLite.
Server	Share databases across processes or machines with remote clients, role-based access control, read-only replicas, and high availability.
Scripting	Use the Babashka pod for fast-starting automation, command-line tools, data jobs, and operational scripts.

There is also an MCP server mode for local AI-tool integration. You can begin with an embedded prototype, move to a shared server as the application grows, and automate it from scripts without rewriting the data model or query language.

Deployment changes. The facts do not.

Start Building

Datalevin 1.0.0 is available now:

Read the new guide at datalevin.org.
Get the complete print or ebook edition of Datalevin: The Definitive Guide.
Explore, use, and contribute to the open-source code on GitHub.
Use Datalevin from Clojure, Java, Python, or JavaScript.

Reaching 1.0 took six years because the goal was never merely to ship another query language or another storage wrapper. The goal was to build one coherent place for application state: simple enough to embed, serious enough to run as a server, expressive enough for relational, graph, document, and logical work, and intelligent enough to become durable memory for the next generation of AI systems.

Thank you to everyone who tested Datalevin, reported issues, contributed code, shared benchmarks, trusted it in production, or simply asked hard questions. You helped turn an ambitious idea into a 1.0 database.

Datalevin 1.0.0 is here. Let us build applications and agents that remember.

SQLite in Production? Not So Fast for Complex Queries

2026-01-27T12:00:00+00:00

There is a growing movement to use SQLite for everything. Kent C. Dodds argues for defaulting to SQLite in web development due to its zero-latency reads and minimal operational burden. Wesley Aptekar-Cassels makes a strong case that SQLite works for web apps with large user bases, provided they don't need tens of thousands of writes per second. Discussions on Hacker News and elsewhere cite companies like Apple, Adobe, and Dropbox using SQLite in production. Even the official SQLite documentation encourages its use for most websites with fewer than 100K hits per day.

These points are fair. The overarching theme is a pushback against automatically choosing complex, client-server databases like PostgreSQL when SQLite is often more than sufficient, simpler to manage, and faster for the majority of use cases. I agree with that framing. The debate has settled into a well-understood set of tradeoffs:

For "SQLite for everything"	Known limitations
Zero-latency reads as an embedded library	Write concurrency limited to a single writer
No separate server to set up or maintain	Not designed for distributed or clustered systems
Reliable, self-contained, battle-tested (most deployed DB in the world)	No built-in user management; relies on filesystem permissions
Fast enough for most human-driven web workloads	Schema migration can be more complex in large projects

These are the terms of the current discussion. But there is an important, often overlooked dimension missing from this framing.

SQLite struggles with complex queries. More specifically, SQLite is not well-suited to handle the kind of multi-join queries that arise naturally in any serious production system. This goes beyond the usual talking points about deployment concerns (write concurrency, distribution, and so on). It points to a system-level limitation: the query optimizer itself. That limitation matters even for read-heavy, single-node deployments, which is exactly the use case where SQLite is supposed to shine.

I have benchmark evidence showing this clearly. This post focuses on join-heavy analytical queries, not on the many workloads where SQLite is already the right choice. But first, let me explain why this matters more than people think.

Multi-join queries are not exotic

A common reaction to discussing multi-join queries is: "I don't write queries with 10 joins." This usually means one of three things: the schema is denormalized, the logic has been moved into application code, or the product is simple. None of these mean the problem goes away.

In any system with many entity types, rich relationships, history or versioning, permissions, and compositional business rules, multi-join queries inevitably appear. They emerge whenever data is normalized and questions are compositional. Here are concrete examples from real production systems.

Enterprise SaaS (CRM / ERP / HR). A query like "show me all open enterprise deals" in a Salesforce-like system touches accounts, contacts, products, pricebooks, territories, users, permissions, and activity logs. Real queries in these systems routinely involve 10-20 joins. Every dimension of the business (customers, ownership, products, pricing, regions, access control, activity statistics) is often normalized into its own table.

Healthcare (EHR). "Patients with condition X, treated by doctors in department Y, prescribed drug Z in the last 6 months, and whose insurance covers that drug" spans patients, visits, diagnoses, providers, departments, prescriptions, drugs, insurance plans, coverage rules, and claims. Exceeding 15 joins is common.

E-commerce and Marketplaces. "Orders in the last 30 days that include products from vendor V, shipped late, refunded, with customers in region R" touches orders, order items, products, vendors, shipments, delivery events, refunds, customers, addresses, regions, and payment methods. Again, 10+ joins.

Authorization and Permission systems. "Which documents can user U see?" requires traversing users, groups, roles, role assignments, resource policies, ACLs, inheritance rules, and organizational hierarchies. This alone can be 12+ joins, sometimes recursive.

Analytics and BI. Star schemas look simple on paper, but real dashboard queries add slowly changing dimensions, hierarchy tables, permission joins, and attribution models. A "simple" dashboard query often hits 6-10 dimension tables plus access control.

Knowledge graphs and semantic systems. "Papers authored by people affiliated with institutions collaborating with company X on topic Y" requires joining papers, authors, affiliations, institutions, collaborations, and topics. Very common in search and recommendation systems.

Event sourcing and temporal queries. Reconstructing the state of an account at a point in time with approval chains requires joining entity tables, event tables, approval tables, history tables, and version joins. Temporal dimensions multiply join counts quickly.

AI / ML feature pipelines. Feature stores generate massive joins. Assembling a feature vector often requires joining user profiles, sessions, events, devices, locations, and historical aggregates. This is why feature stores are expensive.

The pattern is consistent across domains:

Domain	Typical join count
SaaS CRM / ERP	8-20
Healthcare	10-25
Authorization	6-15
BI dashboards	6-12
Knowledge graphs	10-30
Feature pipelines	8-20

Complex joins are not accidental. They emerge from normalized data, explicit relationships, compositional business rules, layered authorization, and historical records. Again, if you don't see many joins in your system, it usually means the schema is denormalized, the logic is in the application layer, or the product hasn't reached sufficient complexity yet. This does not mean the system is better. It often means complexity has been pushed into the application layer, which can add engineering cost without adding real value.

The evidence: JOB benchmark

The Join Order Benchmark (JOB) is a standard benchmark designed specifically to stress database query optimizers on complex multi-join queries [1]. Based on the Internet Movie Database (IMDb), a real-world, highly normalized dataset with over 36 million rows in its largest table, it contains 113 analytical queries with 3 to 16 joins each, averaging 8 joins per query. Unlike synthetic benchmarks like TPC, JOB uses real data with realistic data distributions, making it a much harder test of query optimization.

I ran this benchmark comparing three databases: SQLite (via JDBC), PostgreSQL 18, and Datalevin (an open-source database I build). All were tested in default configurations with no tuning, on a MacBook Pro M3 Pro with 36GB RAM. This is not a tuning shootout, but a look at out-of-the-box optimizer behavior. Details of the benchmark methodology can be found here.

Overall wall clock time

Database	Total time (113 queries)
Datalevin	93 seconds
PostgreSQL	171 seconds
SQLite	295 seconds (excluding 9 timeouts)

SQLite needed a 60-second timeout per query, and 9 queries failed to complete within that limit. The actual total time for SQLite would be substantially higher if these were included. For example, query 10c, when allowed to run to completion, took 446.5 seconds.

Execution time statistics (milliseconds)

Database	Mean	Median	Min	Max
Datalevin	773	232	0.2	8,345
PostgreSQL	1,507	227	3.5	36,075
SQLite	2,837	644	8.1	37,808

The median tells the story: SQLite's median is nearly 3x worse than the other two.

Per-query speedup: Datalevin vs. SQLite

The chart at the top of this post shows the speedup ratio (SQLite time / Datalevin time) for each of the queries on a logarithmic scale (excluding 9 timeouts). Points above the 1x line (10^0) mean Datalevin is faster; points below mean SQLite is faster. The horizontal lines mark 1x, 10x, and 100x speedups.

Several patterns stand out:

The vast majority of points are above the 1x line, often by 10x or more.
For the hardest queries, Datalevin achieves 100x+ speedups. These are precisely the complex multi-join queries where SQLite's optimizer breaks down.
SQLite is rarely faster, and when it is, the margin is small.
The 9 timed-out queries (not shown) would push the ratio even higher.

Where SQLite breaks down

Timeouts. Queries 8c, 8d, 10c, 15c, 15d, 23a, 23b, 23c, and 28c all timed out at the 60-second limit during the benchmark runs. These represent queries with higher join counts where SQLite's optimizer failed to find an efficient plan.

Extreme slowdowns. Even among queries that completed, SQLite was often dramatically slower. Query 9d took 37.8 seconds on SQLite versus 1.6 seconds on Datalevin (24x). Query 19d took 20.8 seconds versus 5.7 seconds. Query families 9, 10, 12, 18, 19, 22, and 30 all show SQLite performing significantly worse, often by 10-50x.

Why SQLite falls behind

SQLite's query optimizer has fundamental limitations for complex joins:

Limited join order search. SQLite uses exhaustive search for join ordering only up to a limited number of tables. Beyond that threshold, it falls back to heuristics that produce poor plans for complex queries.
Weak statistics model. SQLite's cardinality estimation is simpler than PostgreSQL's, which itself has well-documented weaknesses [1]. With fewer statistics to guide optimization, SQLite makes worse choices about which tables to join first and which access methods to use.
No cost-based plan selection for complex cases. For queries with many tables, SQLite's planner cannot explore enough of the plan space to find good join orderings. The result is plans that process orders of magnitude more intermediate rows than necessary.

These limitations are architectural; they are not bugs likely to be fixed in a near-term release. They reflect design tradeoffs inherent in SQLite's goal of being a lightweight, embedded database.

What this means for "SQLite in production"

SQLite is excellent for what it was designed to be: an embedded database for applications with simple query patterns. It excels as a local data store, a file format, and a cache. For read-heavy workloads with straightforward queries touching a few tables, it works extremely well.

But the production systems described above, e.g. CRM, EHR, e-commerce, authorization, analytics, are precisely where SQLite's query optimizer becomes a bottleneck. These are not hypothetical workloads, but the day-to-day reality of systems that serve businesses and users.

The "SQLite in production" advocates often benchmark simple cases: key-value lookups, single-table scans, basic CRUD operations. On those workloads, SQLite does extremely well. But production systems grow. Schemas become more normalized as data integrity requirements increase. Questions become more compositional as business logic matures. And at that point, the query optimizer becomes the bottleneck, not the network round trip to a database server.

Before choosing SQLite for a production system, ask: will our queries stay simple forever? If the answer is no, and it usually is, the savings in deployment simplicity may not be worth the cost in query performance as the system grows.

An alternative approach

In a previous post, I described how Datalevin, a triplestore using Datalog, handles these complex queries effectively. Its query optimizer uses counting and sampling on its triple indices to produce accurate cardinality estimates, resulting in better execution plans. Unlike row stores, where cardinality estimation is notoriously difficult due to bundled storage, a triplestore can count and sample individual data atoms directly.

This approach yields plans that are not only better than SQLite's, but consistently better than PostgreSQL's across the full range of JOB queries. Despite Datalevin being written in Clojure on the JVM rather than optimized C code, it still managed to halve the total query time in the JOB benchmark. The quality of the optimizer's decisions matters more than the raw execution speed of the engine.

For systems that need both deployment simplicity (Datalevin works as an embedded database too) and the ability to handle complex queries as they inevitably arise, a triplestore with a cost-based optimizer offers a practical alternative to either SQLite or a full client-server RDBMS. It is not a silver bullet, but it can deliver SQLite-like operational simplicity without giving up complex-query performance.

If you have different results or have tuned SQLite to handle these queries well, I would love to compare notes. The goal here is not to dunk on SQLite, but to surface a missing dimension in a discussion that often defaults to deployment tradeoffs alone.

References

[1] Leis, V., et al. "How good are query optimizers, really?" VLDB Endowment. 2015.

Triple Store, Triple Progress: Datalevin Posited for the Future

2026-01-19T16:00:00+00:00

In my previous post, Competing for the JOB with a Triplestore, I showed that a triple store, such as Datalevin, can compete with the best row stores on complex relational workloads. Since then, I have rewritten Datalevin's rule engine and improved its storage and query engine. This post focuses on why these matter for the broader goal of using a triple store as a single, flexible data substrate.

One store, many workloads

Our goal is to simplify data storage and access by supporting diverse database use cases and paradigms, because maximal flexibility is the core strength of a triple store. Using one data store for different use cases simplifies and reduces the cost of software development, deployment, and maintenance.

Since 2020, we have been working hard toward this goal of building an easy-to-use and versatile database. I am happy to report that, today, in addition to key-value and relational database features, Datalevin also handles graph queries and deductive logic reasoning tasks, with built-in support for full-text search and vector similarity search as well. All these are seamlessly integrated in a compact package that works in both embedded and server modes.

This post is a guided tour of the progress made so far in the storage engine, the query engine, and the new rule engine.

Triples as the substrate

Datalevin stores data as triples: entity, attribute, value (EAV): the smallest, atomic unit of a data item. This uniform representation is the key that lets one engine handle many shapes of data and many ways of asking questions. A triple store can behave like a relational system when your data is tabular, like a graph system when your data is connected, and like a logic system when you define recursive rules.

The challenge has always been performance. Triple stores have historically been slower than row/column stores. The rest of this post explains how Datalevin addresses this challenge.

Storage

Datalevin uses a fast key-value database library as the storage layer. Specifically, the exceptional read performance of LMDB is the foundation of Datalevin's query performance. Datalevin stores triples with nested indices by leveraging LMDB's DUPSORT capability: the head element of a triple is stored once as the key, and the tail elements are stored as a sorted list of values. This reduces storage overhead and alleviates the data redundancy problem inherent in triple stores.

To further reduce data redundancy, we built our own LMDB fork, DLMDB, which adds page-level prefix compression to the storage. For DUPSORT, prefix compression is applied to both keys and values, resulting in significant storage savings. We also removed the lesser-used VAE index. For a typical reference-heavy (foreign key) Datalog database, the footprint reduction can be over 40%.

Through relentless code optimization, we achieved these savings without incurring excessive read/write overhead. In fact, for the common Datalevin use case of seeking to a key and reading its list of values in full, we obtained a 40% speedup in most cases.

The Datalevin query planner relies heavily on online counting and sampling. To facilitate these operations, we added subtree node count maintenance in DLMDB. These order statistics turn counting and sampling operations from O(n) to O(log n), cutting Datalog query planning time in half. This feature was introduced with minimal write overhead.

Query engine

Datalevin's query planner performs extensive query rewrite passes to optimize performance. For example, predicates are pushed down into index scans so filters execute early rather than after joins; inequality predicates are rewritten into index range scans, and so on.

The planner simplifies the query graph by treating stars as meta-nodes, then applies a Selinger-style dynamic programming algorithm with accurate cardinality estimates. Merge scans collapse star-shaped entity access into a single scan, avoiding redundant joins for attributes belonging to the same entity group. More details on these optimizations can be found in the query engine documentation.

Even with relatively accurate cardinality estimation, occasional bad plans are unavoidable, particularly for tricky joins that link different entities. To reduce the impact of such cases, the cost-based optimizer now considers hash joins as an alternative when input size is large. We also extended the optimizer's coverage to more complex query clauses, such as or-join.

We made the multi-threaded query execution pipeline more robust by handling edge cases in concurrency and adding backpressure. The pipeline now uses its own thread pool to avoid contention with worker thread pools.

Other improvements have focused on usability: "creature-comfort" features like a :having clause, allowing math expressions in :find, specifying sort variables by indices in :order-by, full boolean search expressions for full-text, and so on. These features reduce the amount of custom code needed for post-processing results, and in-database operations are usually more efficient than equivalent work in application code.

Rule engine

Datalevin uses rules to bundle a set of query clauses into named, reusable invocations. As rules can call themselves and other rules, this feature enables recursive logic computation and graph navigation. We wrote a new rule engine that leverages the same cost-based optimizer, allowing Datalevin to serve as an efficient graph database and logic reasoner.

The rule engine uses semi-naive fixpoint evaluation, applies magic-set rewrites where beneficial, and seeds evaluation from outer query bindings so it starts with relevant candidates instead of a blank slate.

In addition, non-recursive rule clauses are inlined into the main query to let the optimizer plan them with index scans and joins. For T-stratified rules, temporal elimination avoids storing unnecessary intermediate results. Detailed information is available in the rule engine documentation.

Benchmarks

We added two new benchmarks to showcase the progress we have made.

Logic workloads: Math Genealogy benchmark

The Math Genealogy benchmark focuses entirely on rule resolution. It is a good stress test for recursive Datalog rules. The dataset contains roughly 256,769 dissertations, 256,767 students, and 276,635 advisor-student relationships. There are four queries in this benchmark. Datalevin's rule engine is very fast on these queries: Q1 (14.4 ms), Q2 (330.9 ms), Q3 (269.6 ms), and Q4 (recursive academic ancestry, 2.9 ms).

By comparison, Datomic takes over 40 seconds on Q4, and Datascript runs out of memory. This query is difficult because recursive ancestry computes a transitive closure: each new level of ancestors can join with every previously found level, which can quickly create a combinatorial explosion of intermediate tuples. Even if the average branching factor is modest (say b=3–5 advisors per student), intermediate results can grow on the order of b^k at depth k. If those tuples are generated repeatedly across branches, we end up materializing large intermediate relations just to discard most of them later.

Why is Q4 so fast in Datalevin? It exploits bound starting points. The semi-naive fixpoint evaluation works off delta relations only, i.e., each iteration only joins newly produced tuples. When a query binds a head argument, the engine also seeds recursion (and, when effective, applies magic-set rewrites) so it only explores the reachable slice of the graph rather than materializing the full closure.

Graph workloads: LDBC SNB benchmark

Graph workloads are where triple stores should shine, but performance is where dedicated graph databases usually try to defend their turf. The LDBC Social Network Benchmark (SNB) is an industry-standard benchmark for interactive graph queries. We implemented the full workload and included a Neo4j implementation for comparison here.

On the SF1 dataset (about 3.2M entities and 17.3M edges), Datalevin is 27x to 620x faster on short interactive queries (pictured above), averaging 48x faster than Neo4j. These short queries are the most commonly encountered workloads in an operational graph database, so performing well on them has significant practical implications.

Some of these short queries (IS2 and IS6) involve unbounded graph traversal, such as finding the root post of a comment. Datalevin handles these with a recursive rule. Thanks to the efficiency of our rule engine, graph navigation performance is stellar.

On complex queries, Datalevin is about 12% faster overall, with some queries (IC6, IC8, IC11) orders of magnitude faster and a few (IC3, IC5, IC9) slower than Neo4j. I am sure Neo4j is extensively tuned for these queries, as it is one of the authors of this industry-standard benchmark. It is remarkable that Datalevin performs so well on these complex graph queries without any specific tuning.

The important observation here is that the same triple store and query engine handle both relational-style joins and graph traversals without needing special cases for either.

Towards the future

A triple store is a flexible substrate. When paired with a cost-based query optimizer and a modern rule engine, it can span relational, graph, and logical reasoning workloads. It can also expand toward richer document workloads without changing the underlying model.

With the current focus on AI systems, a triple store like Datalevin can serve several critical purposes.

An AI agent needs a context graph: entities, facts, relations, constraints, and memories that evolve over time. Keeping that context in one integrated store reduces the impedance mismatch between relational tables, graph edges, embeddings, and documents. Datalevin makes retrieval and grounding first-class: full-text and vector search can pull candidate facts, while Datalog queries and rules can verify, connect, and constrain them. It can also support tool use, where the "tool outputs" are simply more facts to join and reason over. For example, in a RAG pipeline, vector search retrieves candidate snippets, graph relations link them to entities and events, and rules enforce constraints such as provenance or recency.

Datalevin can serve as an agent's memory model, where episodic facts, long-term knowledge, and computed embeddings all live in one place and can be queried together. In that sense, a unified store is to an AI agent what memory is to human cognition: the common ground where different kinds of signals meet to be reasoned over.

Even as AI writes more code, a simple and versatile database remains highly AI-friendly. A context-limited LLM model benefits from a coherent data model and a single query language that covers many use cases. Datalog is truly declarative, meaning there are fewer procedural or implementation details for a model to trip over. That translates to less boilerplate, fewer dialects to remember, and fewer quirks in query semantics, making it more likely that the system handles data correctly. In fact, AI wrote all the Datalevin queries used in the LDBC SNB benchmark mentioned above. Although it is still a relatively niche database, the AI composed queries for Datalevin with ease because the query language is inherently simpler.

Next steps

The recent rule engine rewrite brings us much closer to the 1.0 release milestone. With the addition of high availability, a JSON API, and libraries for other languages, we expect to reach this milestone this year.

After six years of continuous research and development, Datalevin would not be in its current hardened state without the experimentation and production deployment efforts of the Clojure community. I truly appreciate everyone who has used Datalevin and made contributions. The future of simplified data storage and access is near, for human and AI developers alike.

Achieving High Throughput and Low Latency through Adaptive Asynchronous Transaction

2025-02-11T16:15:00+00:00

In my previous post, I demonstrated that Datalevin performs complex queries faster than PostgreSQL. A common reaction is, "Oh, have you tested writing speed? I imagine that when there are indices for everything (as in the case of Datalevin), writing speed could be significantly slower." This post aims to address that write speed concern.

When discussing database write speed, throughput and latency are the two key performance metrics. Throughput is the number of transactions the system can process in a given amount of time—the higher the throughput, the better the system. Latency is the amount of time it takes to process a single transaction from start to finish. In durable databases, this refers to the time between when the transaction starts and when the data is committed and flushed to disk. The lower the latency, the better the system performs.

Obviously, it is desirable to have both high throughput and low latency. However, achieving both simultaneously is often challenging.

Throughput-Latency Trade-Off

To improve throughput, database systems often use batching techniques (grouping many transactions together) to amortize the setup cost across multiple transactions. In particular, the number of expensive disk flush operations can be reduced dramatically through batching. However, waiting to accumulate a batch may increase the latency for individual transactions.

Conversely, processing transactions immediately reduces the wait time for each transaction, lowering latency. Yet handling each transaction independently can prevent the system from fully utilizing available resources, potentially resulting in lower overall throughput.

Asynchronous Transaction

A well-implemented asynchronous transaction model has the potential to improve both throughput and latency simultaneously. Instead of processing a transaction immediately, an asynchronous transaction is placed in a queue—this provides an opportunity to batch it with other transactions, thereby improving overall throughput. Of course, it is still important to ensure that transactions do not wait too long in the queue, which could hurt latency.

Some databases implement asynchronous commit. For example, PostgreSQL’s synchronous_commit parameter can be set to off. In this mode, the system returns transaction success as soon as the transaction is logically completed—before the generated WAL (Write-Ahead Log) records are actually written to disk. However, this implementation compromises durability and consistency guarantees. There is a short window between reporting transaction completion and when the transaction is truly committed. If the system crashes during this risk window, the changes made by that transaction may be lost. PostgreSQL allows users to control the duration of this risk window by setting wal_writer_delay.

Adaptive Asynchronous Transaction

Striking a balance between the advantages of asynchronous processing and the durability guarantees of transactions might seem challenging. In Datalevin, we aim to take advantage of asynchronous transactions without compromising durability. Moreover, having too many tuning parameters goes against our goal of excellent database usability—there is really no good reason for a user to worry about such low-level implementation details.

To achieve these goals, I have implemented an adaptive asynchronous transaction method in Datalevin.

The idea is as follows. First, to maintain transaction durability, the asynchronous transaction method returns a Clojure future, which is only realized once the transaction is fully committed and the data is flushed to disk. The user can dereference the future to determine when the transaction is completed, or block until it is. Alternatively, the method optionally accepts a user-supplied callback function so that the user can be notified when the transaction is complete.

At first glance, these options might seem no different from a synchronous transaction: the user still has to wait until the commit is completed. However, the promise of asynchronous transactions is this: the wait will be shorter than with synchronous transactions. If we can achieve a shorter wait time, we have succeeded.

How can we achieve a shorter wait time? As suggested above, one answer lies in batching transactions.

The challenge now is to find an appropriate batch size that minimizes wait time. As argued earlier, setting a fixed batch size or preset time delay isn’t ideal. Instead, the batch size—or equivalently, the wait time in the queue—should depend on the system load.

When the system is inundated with write requests, a larger batch size is useful for dispatching many requests at once, thus improving throughput. On the other hand, when the system is idle, a write request should be fulfilled immediately, thus minimizing latency. In other words, we want a dynamic batch size that adapts to system load.

Another source of reduced wait time is improved concurrency. While waiting for an I/O operation to complete, other tasks can use the CPU. Proper design allows this overlapping of computation and I/O to effectively “hide” I/O wait times.

The most direct way to achieve this is through a simple mechanism [1]: when processing a queued asynchronous write event, the system checks its event queue to see if there are other similar events waiting. If so, the system simply ignores the duplicate event and moves on to the next one. Otherwise, it submits a task to a worker thread pool to process the queued asynchronous transactions in a batch.

This mechanism exhibits the adaptive behavior described above. Under a heavy load, the batch size increases; when the load is low, the batch size is small. The available system resources are effectively utilized to optimize both throughput and latency outcomes—without compromising durability.

Metric

In order to measure how well we have achieved our goal, we need a metric that combines throughput and latency. Combining these two metrics into a single number is challenging because they measure different aspects of performance and have different units. There may not be a perfect answer, but here are a few ideas.

One simple composite is to compute a ratio such as Throughput / Latency. A higher value implies that the system is processing more transactions quickly (i.e., achieving high throughput and low latency). However, if throughput is measured in transactions per second and latency in seconds per transaction, the ratio has units of transactions squared per second, which makes its interpretation less obvious.

One way to refine this formula is to introduce a target latency required by a particular use case. For example, for low-latency applications that expect sub-millisecond database transactions, the target latency might be 1 millisecond. For applications that can tolerate slightly higher latency, the target might be 10 milliseconds.

With a target latency in place, we can define an effective “good throughput” for a given use case. One simple approach is to combine throughput with the fraction of operations finishing within that target latency. For instance, one might define a metric like this:

Effective Throughput = Actual Throughput × (Target Latency / Actual Latency)

This metric rewards systems that process many transactions and keep them within acceptable latency boundaries. If the latency exceeds the target, the effective throughput metric drops, reflecting a degraded user experience.

Effective Throughput (ET) is a use-case–dependent metric that is simple to interpret. The higher the ET, the better the system. It is also easy to calculate, as its value is proportional to the target latency. For example, the ET for a 10-millisecond target is 10 times that for a 1-millisecond target. Thus, we really only need to calculate the ratio of throughput to latency (i.e., for a target latency of 1 unit) and then extrapolate to other target latencies. For the analysis below, we will adopt ET.

Benchmark

To address the write-speed concerns of Datalevin, we compare Datalevin with SQLite—the most widely used embedded database, renowned for its fast writes. Although Datalevin can be used in a client/server mode, for this benchmark we use it as an embedded database.

Using the same dataset, we compare four transaction conditions: Datalevin Default, Datalevin Async, SQLite Default, and SQLite WAL. These are all fully durable transaction methods—a transaction is considered complete only after its data is fully flushed to disk.

Database transaction benchmark performance is highly sensitive to hardware and the operating system; the numbers can vary widely between different machines. In this benchmark run, we used a machine with a 2016 Intel Core i7-6850K CPU @ 3.60GHz (6 cores), 64GB RAM, and a Samsung 860 EVO 1TB SSD. This machine performs around the middle of the pack in Today's CPU Benchmark, so it is reasonably representative.

The code and detailed description of the benchmark can be found in our GitHub repository.

Pure Write

The benchmark has two tasks. The first is a pure write task, where each write consists of an entity (row) with two attributes (columns). Every 10,000 writes, we measure the throughput and latency. In addition to individual writes, we also measure performance when writing data in batches—testing batch sizes of 10, 100, and 1000.

Using the Effective Throughput (ET) metric described above, the bar chart at the top of this post shows the pure write task results. (Note: the Y axis is logarithmic.)

When the batch size is 1 (i.e., writing a single entity at a time), Datalevin Async achieves the best ET. It is several orders of magnitude higher than under other write conditions—not only is the raw throughput high (16,829.2 writes per second), the latency is low (0.1 milliseconds) as well. When writes are batched, the ET of Datalevin Async slightly decreases due to the latency increasing faster than throughput.

Datalevin Default performance peaks at a batch size of 100; however, at that batch size, SQLite’s performance outperforms it.

SQLite Default performs very poorly for individual writes, though SQLite WAL is much better. Both benefit significantly from increased batch sizes, with SQLite Default benefiting the most.

In general, Datalevin's write performance is more stable and less sensitive to variations in batch size compared to SQLite. Because Datalevin performs indexing at transaction time, its write performance does not benefit as much from increased batch sizes.

For an operational database, each transaction normally contains a small number of entities (or rows), as recommended by industry best practices [2]. In these online transaction processing (OLTP) workloads, Datalevin is expected to perform better than SQLite.

For use cases involving the bulk loading of data, Datalevin provides init-db and fill-db functions that bypass the expensive transactional processes and are more appropriate.

Mixed Read/Write

After one million entities (rows) have been loaded, the second task alternates between reading a row and writing a row until one million reads and one million writes have been performed. For this task, we report the results using Linux’s time command.

The first chart shows wallclock time, while the second shows user and system CPU time.

For the mixed read/write task, Datalevin Default is much faster than SQLite Default, and Datalevin Async is much faster than SQLite WAL, while SQLite WAL outperforms Datalevin Default.

Regarding CPU time, the differences among the various conditions are small, indicating that the underlying amount of work is not hugely different.

Notice that in the three synchronous conditions—Datalevin Default, SQLite Default, and SQLite WAL—most of the time is spent waiting for I/O, with CPU times being relatively small compared to the wallclock time. Datalevin Async is different; its total CPU time (227.89 seconds) is actually greater than its wallclock time (111.04 seconds), indicating effective utilization of multicore processing and an apparent hiding of I/O wait times. This result confirms our hypothesized advantage of asynchronous transactions.

Conclusion

We can now answer the question about Datalevin's write speed: it performs better than SQLite under OLTP workloads. While the default synchronous write mode performs at a respectable level, Datalevin truly shines with asynchronous transactions, achieving both high throughput and low latency without compromising transaction durability.

Reference

[1] Nathan Marz, "2.5x Better Performance: Rama vs. MongoDB and Cassandra", April 2024. (https://blog.redplanetlabs.com/2024/04/25/better-performance-rama-vs-mongodb-and-cassandra/)

[2] Gray, J., & Reuter, A. Transaction Processing: Concepts and Techniques. Morgan Kaufmann Publishers, 1993.

Competing for the JOB with a Triplestore

2024-09-10T18:49:00+00:00

This is a post about Datalevin, an open-source database system that I have been building since 2020. Although JOB here is really an acronym for Join Order Benchmark, a benchmark for complex database queries, I intend to build up Datalevin to compete for the same kind of jobs usually held by a RDBMS. I hope this title is received well and not seen as just a pun :-). This post describes my journey in making Datalevin competitive in the JOB benchmark, where it outperformed PostgreSQL by a small margin.

Datalevin is a triplestore, which means it stores data in units of three elements known as triples. Specifically, the three elements are entity (E), attribute (A) and value (V). For example, to store a piece of information: a person is named "John Smith", E would be the entity id number of this person, A would be an attribute named :person/name , and V would be "John Smith", the attribute value of the entity.

Unlike what the Wikipedia page of triplestore seems to suggest, a triplestore is not necessarily about Semantic Web use cases. Datalevin is a general purpose database system that follows an approach pioneered by Datomic, and uses its flavor of Datalog as the query language.

Why Datalog?

The flavor of Datalog used in Datalevin can be considered a modern alternative to SQL.

There have been many articles discussing the limitations of SQL as a database query language, such as Against SQL and We can do better than SQL, and I am sympathetic towards those views and recommend readers to look into them.

One point I would like to add, from a point of view of human computer interaction, is related to the historical context of SQL [1]. As one of those ALL CAP languages (e.g. COBOL), SQL was from an era when enthusiastic computer scientists believed that structured English could be a good human interface to computers. The rationale was that business people could be taught to use such languages to communicate with computers. We all know how that turned out: business people are allergic to any kind of formal languages, i.e. code, so these languages still end up as tools for programmers only. However, structured English makes for poor programming languages, because it is difficult to parse, compose or extend. SQL is the most stubborn left-over from those ill-considered experiments, perhaps due to the fact that data often last longer than programs.

Datalog used in Datalevin does not have such historical baggage. Let us look at an example:

;; query about persons who are adult males, whose name ends with Smith

[:find ?name
 :where
 [?person :person/age ?age]
 [?person :person/name ?name]
 [(like ?name "%Smith")]
 [(<= 18 ?age)]]

Similar to SQL's SELECT and WHERE, this flavor of Datalog has explicit :find and :where sections, so it is easier to grok than the original Datalog syntax from Prolog, which read like math expressions. Unlike SQL though, this query language is represented as a data structure in EDN format, which is amicable to programmatic manipulation, just like JSON.

Each where clause is delimited by a pair of [ ]. All the where clauses are AND'ed together, i.e. they all must evaluate to be true for the query to return non-empty results. There are two types of where clauses in this query: triple patterns and predicates.

Just like a triple, a triple pattern consists of Entity (E), Attribute (A) and Value (V), in that order. Any of the three elements could be a variable, which is a user defined symbol starting with a ?. For example, triple pattern [?person :person/age 18] says that there are entities that we name with a variable ?person, and they have an attribute :person/age with the value 18. Triple patterns must match with the data, so that variables can be bound with values from the database. For this pattern, ?person would be bound with entity IDs of all persons aged 18 in the database.

Predicates are boolean functions, and they are enclosed in a pair of ( ). The first element inside the parentheses is the function name, e.g. like, <=, and so on, and the rest are the function arguments. All the predicates must evaluate to be true for the query to return non-empty results.

That's it, the syntax of Datalevin's Datalog. There are neither complex syntactic rules, nor hundreds of reserved words.

More importantly, Datalog is very high level and truly declarative. It does not expose low level technical details of database operations to the users, nor requires the users to speak database jargons or be a relational algebra expert to use the language well. For example, the user is not required to understand the concept of joins, let alone the many different types of joins: inner, left, right, full, self joins, etc.

Joins are implicit in Datalog and are completely oblivious for the users. All users need to focus on are the logic relationships in their data, the rest is left for the database system to figure out. Datalog greatly simplifies the writing of complex queries. This is perhaps the main strength of this query language. My experience in onboarding new users has been that it normally took people less than half an hour to become a competent Datalog query writer.

Regardless the language level differences, Datalog in Datalevin is still backed up by a computational substrate based on relational algebra. It just treats relational algebra as underlying implementation details that users should not be concerned with.

Why a triple store?

When people talk about relational databases, they often assume it's a row store or a column store, as if relational algebra has something to do with how a table is stored. But, it doesn't. As a mathematical theory, relational algebra does not specify how a relation should be stored in computers. Therefore, in addition to storing a relation as rows or columns, storing it as triples are equally sound, if not better.

For example, for the data set above, a row store would store them as two tables.

Storing a table as triples means storing the table as a list of table cells, i.e. as a list of the smallest identifiable units of the table.

Storing table cells as rows or columns are just two concrete ways of bundling the table cells together for efficiency purpose, so that column identifiers or row identifier would not need to be repeated. That is to say, they are optimizations. So, the question is, are these "premature optimizations"?

I argue that they are. One problem with fixating the storage to either a row or a column format, is that it impedes querying the data later on. Specifically, it makes the work of query optimizer unnecessarily hard.

Essentially, both row and column storage are place oriented: table cells are identified by positions, rather than by explicit identifiers. One consequence is that missing data cells still have to be represented positionally, i.e. they need to take up places. RDBMS normally store these as special null values. This is what complicates the collection of statistics of table cells, which a query optimizer demands: it needs to know how many non-null data points it is dealing with under all kinds of conditions. The place based storage does not have a cheap way of providing this information, because any position could hold a null value, or not.

To work around this problem, RDBMS resorts to expensive and complicated processes to collect various approximations of exact data counts, e.g. histograms. Still, the lack of direct information often forces the optimizer to make unrealistically simplistic statistical assumptions about data distribution, e.g. independence, uniformity, and so on. This so called "cardinality estimation" problem is such a notoriously hard topic in database research that it is sometimes called "Achilles Heel" of database query processing [2].

In a triplestore, each table cell is identified explicitly by both row and column, e.g. E is the row, A is the column, and V is the value. A triple alone has complete information about a piece of data. Hence, a triple is called a datom (data atom) in this flavor of Datalog. Some simplifications naturally arise in such a triplestore.

In a triplestore, a missing table cell is simply missing from the storage: null is not stored, and absence means null. To remove a piece of data, one actually removes it from the storage, instead of setting some values to null. This is a critical simplification that enables easy and accurate counting of non-null data in the database.

Another simplification is that the entire database is essentially just one big table. E and A are all globally scoped. In Datalevin, E is represented as a 64 bit integer, and A a 32 bit integer. Database schema specifies only information about each attribute, e.g. its id, data type, uniqueness, and so on.

;; Datalevin schema for the example data above

{:person/name   {:db/aid 1 :db/valueType :db.type/string}
 :person/age    {:db/aid 2 :db/valueType :db.type/long}
 :person/school {:db/aid 3 :db/valueType :db.type/ref}
 :school/name   {:db/aid 4 :db/valueType :db.type/string}
 :school/city   {:db/aid 5 :db/valueType :db.type/string}}

Information about individual tables are no longer necessary. Name spaced attribute names still allow grouping of attributes, e.g. :person/name, person/age are attributes of person entity, but this is just a convention, not a physically separated table. Tables dedicated to joins also disappear, as they are simply handled by attributes with entity id as values (i.e. of reference data type: :db.type/ref).

The atomic nature of a datom allows maximum flexibility, as these datoms can be arranged in ways that facilitate querying them. For example, if datoms are ordered by E first, then A, then V, one essentially gets a row major storage that has the similar properties of a row store. If datoms are ordered by A first, one gets a column major storage that has the same advantage of a column store. Tripestores often store data in both orderings, so that essentially all data are indexed by default. Not having to manage indices can save some headache.

Obviously, the storage of triplestores is redundant, as row identifiers and column identifiers are repeated. If storing both row major and column major data ordering, even more storage spaces are taken. More data means higher I/O demand, and slower querying may be resulted. This is perhaps the main challenge that has prevented the widespread use of triplestores.

I argue that this data storage redundancy problem is easier to mitigate than the hard problem of cardinality estimation resulted from bundling table cells as rows or columns. For example, prefix compression could be used to minimize the repetition. Datalevin utilizes this strategy by storing data in a two level nesting schema: AV is nested underneath E in EAV ordering, and E underneath AV in AVE ordering. Obviously, three level nesting can further reduce data redundancy.

Critically, with unbundled table cells in triplestores, the problem of cardinalty estimation can be solved with straightforward counting or sampling. I argue that this advantage is perhaps a good reason to use a triplestore. As a demonstration, Datalevin employed this simple strategy of counting and sampling to achieve better performance than PostgreSQL in JOB benchmark.

Join Order Benchmark (JOB)

This benchmark aims to test a database query optimizer's capacity to handle complex queries that involve many joins [3]. Based on Internet Movie Database, 113 analytical SQL queries were developed to have between 3 and 16 joins, with an average of 8 joins per query. These SQL queries were translated into the equivalent Datalevin queries, which were verified to produce exactly the same results as corresponding SQL ones.

The tests were conducted on a MacBook Pro Nov 2023, Apple M3 Pro chip, 6 performance cores and 6 efficiency cores, 36GB memory, and 1TB SSD disk.

On average, Datalevin is about 1.3 times faster than PostgreSQL 16 in this benchmark. More details can be found here.

Analysis

Given that Datalevin is written in Clojure — a functional programming language on JVM, not widely known for speed, and PostgreSQL is written in highly optimized C code, it's clear that the query optimizer of Datalevin generates better execution plans than that of PostgreSQL on average.

As I will show with an example, Datalevin's better plan is mainly achieved by having more accurate estimate of result sizes at each execution step, corroborating with the point argued above regarding the superiority of triplestores.

Query 20a of JOB is what PostgreSQL did the worst compared with Datalevin. It took PostgreSQL over a minute to execute its plan whereas Datalevin took about a second. Let us look at the SQL query first.

-- query 20a

SELECT MIN(t.title) AS complete_downey_ironman_movie
FROM complete_cast AS cc,
     comp_cast_type AS cct1,
     comp_cast_type AS cct2,
     char_name AS chn,
     cast_info AS ci,
     keyword AS k,
     kind_type AS kt,
     movie_keyword AS mk,
     name AS n,
     title AS t
WHERE cct1.kind = 'cast'
  AND cct2.kind LIKE '%complete%'
  AND chn.name NOT LIKE '%Sherlock%'
  AND (chn.name LIKE '%Tony%Stark%'
       OR chn.name LIKE '%Iron%Man%')
  AND k.keyword IN ('superhero',
                    'sequel',
                    'second-part',
                    'marvel-comics',
                    'based-on-comic',
                    'tv-special',
                    'fight',
                    'violence')
  AND kt.kind = 'movie'
  AND t.production_year > 1950
  AND kt.id = t.kind_id
  AND t.id = mk.movie_id
  AND t.id = ci.movie_id
  AND t.id = cc.movie_id
  AND mk.movie_id = ci.movie_id
  AND mk.movie_id = cc.movie_id
  AND ci.movie_id = cc.movie_id
  AND chn.id = ci.person_role_id
  AND n.id = ci.person_id
  AND k.id = mk.keyword_id
  AND cct1.id = cc.subject_id
  AND cct2.id = cc.status_id;

The PostgreSQL EXPLAIN ANALYZE output for this query:

                                                                                          QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=21.78..2474.54 rows=1 width=17) (actual time=17052.871..72628.822 rows=33 loops=1)
   ->  Nested Loop  (cost=21.35..2474.09 rows=1 width=21) (actual time=17052.352..72626.880 rows=33 loops=1)
         ->  Nested Loop  (cost=20.93..2473.63 rows=1 width=25) (actual time=17042.110..72584.290 rows=1314 loops=1)
               ->  Nested Loop  (cost=20.50..2473.17 rows=1 width=29) (actual time=1.352..35903.879 rows=87986607 loops=1)
                     Join Filter: (ci.movie_id = t.id)
                     ->  Nested Loop  (cost=20.06..2471.42 rows=1 width=33) (actual time=1.124..3346.138 rows=978322 loops=1)
                           ->  Nested Loop  (cost=19.63..2469.68 rows=1 width=25) (actual time=0.703..2394.487 rows=28583 loops=1)
                                 ->  Nested Loop  (cost=19.48..2469.50 rows=1 width=29) (actual time=0.688..2362.288 rows=73560 loops=1)
                                       ->  Nested Loop  (cost=19.05..2467.72 rows=1 width=4) (actual time=0.444..41.257 rows=85941 loops=1)
                                             ->  Hash Join  (cost=18.89..2462.46 rows=190 width=8) (actual time=0.416..22.960 rows=135086 loops=1)
                                                   Hash Cond: (cc.status_id = cct2.id)
                                                   ->  Seq Scan on complete_cast cc  (cost=0.00..2086.86 rows=135086 width=12) (actual time=0.365..6.616 rows=135086 loops=1)
                                                   ->  Hash  (cost=18.88..18.88 rows=1 width=4) (actual time=0.035..0.035 rows=2 loops=1)
                                                         Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                                         ->  Seq Scan on comp_cast_type cct2  (cost=0.00..18.88 rows=1 width=4) (actual time=0.027..0.028 rows=2 loops=1)
                                                               Filter: ((kind)::text ~~ '%complete%'::text)
                                                               Rows Removed by Filter: 2
                                             ->  Memoize  (cost=0.16..0.18 rows=1 width=4) (actual time=0.000..0.000 rows=1 loops=135086)
                                                   Cache Key: cc.subject_id
                                                   Cache Mode: logical
                                                   Hits: 135084  Misses: 2  Evictions: 0  Overflows: 0  Memory Usage: 1kB
                                                   ->  Index Scan using comp_cast_type_pkey on comp_cast_type cct1  (cost=0.15..0.17 rows=1 width=4) (actual time=0.119..0.119 rows=0 loops=2)
                                                         Index Cond: (id = cc.subject_id)
                                                         Filter: ((kind)::text = 'cast'::text)
                                                         Rows Removed by Filter: 0
                                       ->  Index Scan using title_pkey on title t  (cost=0.43..1.78 rows=1 width=25) (actual time=0.027..0.027 rows=1 loops=85941)
                                             Index Cond: (id = cc.movie_id)
                                             Filter: (production_year > 1950)
                                             Rows Removed by Filter: 0
                                 ->  Index Scan using kind_type_pkey on kind_type kt  (cost=0.15..0.17 rows=1 width=4) (actual time=0.000..0.000 rows=0 loops=73560)
                                       Index Cond: (id = t.kind_id)
                                       Filter: ((kind)::text = 'movie'::text)
                                       Rows Removed by Filter: 1
                           ->  Index Scan using movie_id_movie_keyword on movie_keyword mk  (cost=0.43..1.29 rows=46 width=8) (actual time=0.024..0.031 rows=34 loops=28583)
                                 Index Cond: (movie_id = t.id)
                     ->  Index Scan using movie_id_cast_info on cast_info ci  (cost=0.44..1.31 rows=35 width=12) (actual time=0.001..0.028 rows=90 loops=978322)
                           Index Cond: (movie_id = mk.movie_id)
               ->  Index Scan using char_name_pkey on char_name chn  (cost=0.43..0.46 rows=1 width=4) (actual time=0.000..0.000 rows=0 loops=87986607)
                     Index Cond: (id = ci.person_role_id)
                     Filter: ((name !~~ '%Sherlock%'::text) AND ((name ~~ '%Tony%Stark%'::text) OR (name ~~ '%Iron%Man%'::text)))
                     Rows Removed by Filter: 0
         ->  Index Scan using keyword_pkey on keyword k  (cost=0.42..0.45 rows=1 width=4) (actual time=0.032..0.032 rows=0 loops=1314)
               Index Cond: (id = mk.keyword_id)
               Filter: (keyword = ANY ('{superhero,sequel,second-part,marvel-comics,based-on-comic,tv-special,fight,violence}'::text[]))
               Rows Removed by Filter: 1
   ->  Index Only Scan using name_pkey on name n  (cost=0.43..0.45 rows=1 width=4) (actual time=0.059..0.059 rows=1 loops=33)
         Index Cond: (id = ci.person_id)
         Heap Fetches: 0
 Planning Time: 28.727 ms
 Execution Time: 72629.051 ms
(50 rows)

As can be seen, the catastrophic running time is mainly due to vastly underestimated result join sizes, leading to sub-optimal join order. For instance, the planner estimated that the hash join of complete_cast and comp_cast_type would produce 190 rows but actually got 135086 rows. Similar under-estimation is persisted and its effect accumulated, eventually reaching a staggering 1 vs. 87986607 discrepancy with actual result size.

These underestimations are due to the strong statistical assumptions its estimator makes: value uniformity and column independence. However, the actual value distribution is highly skewed and the columns are highly correlated with one another, as it is often the case in real world datasets. I would like to point out that this problem is not unique to PostgreSQL, but in fact common in all row based RDBMS [3]. These strong assumptions are necessary because more direct information is not available in row stores.

Datalevin does not rely on such strong statistical assumptions to estimate join result size. Instead, it simply counts or samples to obtain the base selectivity ratios. Then, for each pair of base relations, it actually executes the joins using the samples, and thus obtains the pair-wise join selectivity ratios. These ratios are simply used in estimation of join sizes in later joins. The only assumption it makes is that the selectivity ratios of pair-wise joins are similar to those of later joins. If the samples are representative, this assumption would not be too off base.

For query 20a, the corresponding Datalog query is the following:

;; query 20a

 [:find (min ?t.title)
  :where
  [?cct1 :comp-cast-type/kind "cast"]
  [?cct2 :comp-cast-type/kind ?cct2.kind]
  [(like ?cct2.kind "%complete%")]
  [?chn :char-name/name ?chn.name]
  [(not-like ?chn.name "%Sherlock%")]
  [(or (like ?chn.name "%Tony%Stark%") (like ?chn.name "%Iron%Man%"))]
  [?k :keyword/keyword ?k.keyword]
  [(in ?k.keyword ["superhero", "sequel", "second-part",
                  "marvel-comics", "based-on-comic",
                  "tv-special", "fight", "violence"])]
  [?kt :kind-type/kind "movie"]
  [?t :title/production-year ?t.production-year]
  [(< 1950 ?t.production-year)]
  [?t :title/kind ?kt]
  [?mk :movie-keyword/movie ?t]
  [?ci :cast-info/movie ?t]
  [?cc :complete-cast/movie ?t]
  [?ci :cast-info/person-role ?chn]
  [?ci :cast-info/person ?n]
  [?mk :movie-keyword/keyword ?k]
  [?cc :complete-cast/subject ?cct1]
  [?cc :complete-cast/status ?cct2]
  [?t :title/title ?t.title]]

The :plan portion of the explain output is the following:

  [[{:steps       ["Initialize [?k ?k.keyword] by range [[[:closed \"based-on-comic\"] [:closed \"based-on-comic\"]] [[:closed \"fight\"] [:closed \"fight\"]] [[:closed \"marvel-comics\"] [:closed \"marvel-comics\"]] [[:closed \"second-part\"] [:closed \"second-part\"]] [[:closed \"sequel\"] [:closed \"sequel\"]] [[:closed \"superhero\"] [:closed \"superhero\"]] [[:closed \"tv-special\"] [:closed \"tv-special\"]] [[:closed \"violence\"] [:closed \"violence\"]]] on :keyword/keyword."],
     :cost        17,
     :size        8,
     :actual-size 8}
    {:steps       ["Merge ?mk by reverse reference of :movie-keyword/keyword."
                   "Merge [?t] by scanning [:movie-keyword/movie]."],
     :cost        41,
     :size        35549,
     :actual-size 35548}
    {:steps       ["Merge ?cc by equal values of :complete-cast/movie."
                   "Merge [?cct2 ?cct1] by scanning [:complete-cast/status :complete-cast/subject]."],
     :cost        147569,
     :size        11909,
     :actual-size 12810}
    {:steps       ["Merge ?ci by equal values of :cast-info/movie."
                   "Merge [?chn ?n] by scanning [:cast-info/person-role :cast-info/person]."],
     :cost        196991,
     :size        472311,
     :actual-size 476091}
    {:steps       ["Merge [?chn.name] by scanning [:char-name/name]."],
     :cost        2180697,
     :size        472,
     :actual-size 33}
    {:steps       ["Merge [?t.title ?kt ?t.production-year] by scanning [:title/title :title/kind :title/production-year]."],
     :cost        2184661,
     :size        472,
     :actual-size 33}
    {:steps       ["Filter by predicates on [:comp-cast-type/kind]."],
     :cost        2186643,
     :size        472,
     :actual-size 33}
    {:steps       ["Filter by predicates on [:kind-type/kind]."],
     :cost        2188625,
     :size        472,
     :actual-size 33}
    {:steps       ["Merge [?cct2.kind] by scanning [:comp-cast-type/kind]."],
     :cost        2190607,
     :size        472,
     :actual-size 33}]]

As can be seen, the estimated join sizes are close to the actual sizes. Particularly for the initial steps, the estimations are almost exact. For data sizes that smaller than a user configured threshold, actual counting of the data is performed, while sampling was performed for larger data sizes. Both are performed under query specific conditions.

Compared with the tree shape of the PostgreSQL's plan, the plan in Datalevin is linear, because we only consider left-deep join tree. We join one base relation at a time in order to take advantage of the accurate counting and sampling of base relations.

Counting and sampling under query conditions are straightforward in Datalevin, thanks to the nested triple storage. Applicable value predicates are translated into range queries in AVE ordering of triples. When both A and V are known, obtaining the count is a constant time operation. When V is a range, the count is just the sum of the counts in range; The only missing information is the total count of A, which is easily maintained during transaction.

More details of the Datalevin query planner and execution engine can be found here.

Summary

With Datalevin, I have demonstrated that a triplestore has some unique advantages compared with row or column based databases. It helps with solving the difficult problem of cardinality estimation, while its shortcomings can be easily mitigated by nested storage of triples. Paired with a Datalog query language, it makes a promising choice as a modern alternative to SQL RDBMS, as it is much more ergonomic.

References

[1] Chamberlin, D. and Boyce, R.. "SEQUEL: A Structured English Query Language", Proc. ACM SIGFIDET Workshop on Data Description Access and Control. 1974.

[2] Leis, V., et al. "Cardinality Estimation Done Right: Index-Based Join Sampling." Cidr. 2017.

[3] Leis, V., et al. "How good are query optimizers, really?." VLDB Endowment. 2015.

Why Clojure is not widely adopted like mainstream languages?

2022-06-07T06:35:41+00:00

Someone asked on Reddit: Why Clojure is not widely adopted like mainstream languages?

Here's my answer:

You probably should have flipped the question, because the vast majority of programming languages are not mainstream.

The more insightful question is: "Why and how did a language X become mainstream?"

From the answers to this flipped question, we may induce some patterns that other languages could try to imitate.

Let's give it a try, by looking at the most popular languages as of today:

Python: Some big corporations adopted it early on, e.g. Google; some well-connected people started to prompt it as the first language, then schools started following the suggestion; now it enjoys dominance in machine learning - the most hyped field.
C: A readable assembly, so almost all operating systems are written in it; you must use it if you want to interface with hardware; all students in the last generation learnt it in school.
Java: Heavily prompted by its corporate creator that enjoyed a high status among developers; billed as something solving most problems at that time; consequently widely adopted by other big corporations as well; so it was taught in schools everywhere.
C++: billed as the next step of C, due to the wise choice of its name; heavily prompted by tooling companies such as Microsoft, for it was so complex, their tooling were required; all the "serious" computer science students are then proud of their skills in C++, because only they can master it, non-CS students would not be able to.
C#: Microsoft's answer to Java. Microsoft betted the company on it.
Visual Basic: non-programmer's only option for a very long time.
Javascript: the Web's only frontend option for a very long time.

So the tricks to popularity seems to be, in order of importance:

Big company backing
Niche dominance
School adoption

Clojure's only hope is 2. Maybe 1, if one of the companies using Clojure becomes wildly successful. Then 3 will follow.

What do you think?

T-Wand: Beat Lucene in Less Than 600 Lines of Code

2021-11-05T14:49:00+00:00

Here is the story. I am adding full-text search capability to Datalevin, a Datalog database that we open sourced last year. For this task, I have decided to write a search engine from scratch instead of using an existing search library. Here are some rationales for this decision. Today I finished the main work of the search engine, and ran some benchmark comparison with Apache Lucene, the venerable Java search library, and found that the Datalevin search engine is 75% faster on average than Lucene, while 3 times faster at the median point. Since Lucene is at such a dominant position in full-text search, I think it might be of broad interest to write about it.

Yes, it is true. The search engine is less than 600 lines of Clojure code.

Lucene is Hard to Beat

Lucene has over 20 years of history, with perhaps hundreds of man-year of work behind her development. Any search engine coming out with better numbers than Lucene is suspicious of dishonesty, incompetence in benchmarking, or both. I will not name names, but a Google search should show some samples of these. Obviously, I would not want to add Datalevin to such a hall of shame. Please do report any errors in my benchmarking.

Better Search Algorithm

To beat Lucene, it is obviously not enough to optimize code, as Lucene is highly optimized to the last fine details. For example, Lucene has multiple versions of Priority Queue implementations, each customized to a different use case. It is insane.

So, to beat Lucene, having better algorithm is pretty much a requirement. Luckily, I did come up with a better search algorithm, which I call T-Wand.

By "better", I mean it in two senses.

Better Relevance

How many time have you felt frustrated that the search engine gave you results that you know are worse than what you should be getting?

I know there are some documents containing all these words, why are they not here?

"Here" means the first page, or being in the "top-K" results in research parlance. Full text search engines, like Lucene, are often optimized to return a limited number (K) of good results as quickly as possible. However, what is a "good" result? This is where the concept gets muddier. It is no longer a purely technical problem, but more of a user experience problem. In research parlance, this is called "relevance".

As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that:

A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of user query terms.

This makes perfect sense. Right?

Right. Unfortunately, most search engines, including Lucene, do not do that.

Most search engines use something called a vector space model, where both user queries and documents are reduced to vectors (i.e. a fixed number of numbers). That is to say, the search engines are not looking at the meanings of the queries or the documents. Instead, they turned them both into some numbers. The search problem, is reduced to a problem of finding the similarity between the numbers representing the query and the numbers representing the documents.

An often used similarity measure, is to treat these numbers as coordinates in some kind of space. Now both a query and a document become vectors (or points) in this space. And if you remember any of the high school math, you will know that one can calculate the angle between a query vector and a document vector. This angle is the similarity search engines use to rank the relevance of documents to the query. The smaller is the angle, the higher ranking is a document.

This is an elegant model. However, as you can see, this vector space model does not explicitly require a higher ranking document to contain more query terms than a lower ranking one. The results often come out violating the above requirement, inducing user frustrations. This is a well-known problem. For example, Lucene 3.5.0 (when they were still humble enough to write this?) introduced its scoring features with this paragraph:

Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on java-user@lucene.apache.org to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms.

T-Wand wants to change that.

Better Search Speed

T of T-Wand stands for Tiered. As you might have guessed, our search algorithm divides the document collections into tiers. The first tier are those documents containing all n user query terms, the second tier are those containing n-1 user query terms, so on and so forth.

With this division, we no longer need to consider the whole document collection as the potential result candidates. Instead, only those in the current tier are. With much less work to do, the search speed is going to be better.

T-Wand Algorithm

First, let me introduce the WAND algorithm.

WAND

Did I say Lucene is fast? Lucene is very fast, because it uses one of the state of art search algorithms, WAND [1]. Here's how WAND works.

It cheats.

Well, any sufficiently advanced algorithm looks like cheating. WAND is no exception. Basically, it skips a large portion of document collection, and it skips them safely, meaning the results would be the same if one exhaustively does the full computation without skipping. It is able to safely skip documents by using two tricks.

The first trick is the most ingenious. I still do not know how my former colleagues at IBM Research came up with it. My hat's off to them. Let me steal a picture from a followup article [2] to illustrate this.

As can be seen, the 4 rows are the document ids of 4 query terms, and four iterators are walking the document ids from left to right. At each step of the iteration, WAND arranges the rows such that the current document ids of the rows are sorted from lower to higher.

Now, starting from the first row, we sum up a maximal "goodness" score of that row's documents, as soon as the "goodness" score pass a certain threshold, we stop at that row. Say, this row is the 3rd one. At this point, we can teleport the first two rows's iterators to the document ids where the 3rd one is at, skipping every documents in between. This skipping is safe, because, by definition, the document pointed by the 3rd iterator is the first one that pass the threshold. Those before it would not have passed, because the "goodness" scores we use for each row are the maximal of those rows. There could not possibly be anything larger in those skipped documents.

As we have alluded to, the second trick, is this idea of using a hypothetical maximal possible score to filter out potential candidates without having to fully examine them: "we already gave you all the slack possible, yet you still cannot pass the bar, so we can safely kick you out without having to look at you in details". This is a fairly general idea that is used in many algorithms. For example, I would consider A* algorithm is in the same spirit. This reminds me also of a Chinese strategic doctrine, "料敌从宽“, it means to estimate the strength of the opponent in the most generous term possible, and to plan accordingly.

In WAND, the threshold used to filter out documents is the current lowest score of those documents that have made into the top-K. This threshold becomes more difficulty to pass as the algorithm proceeds, filtering out greater proportion of documents as it goes.

What's New in T-WAND

So it seems to be obvious to apply the same WAND algorithm in each tier. But that would not be efficient, because now you are going to have multiple passes over the document collection, doing a lot of wasted work.

Another idea came into play, which was actually the first idea I tried. This idea was inspired by an idea in another research field, approximate string matching [3]. In retrospect, this idea has the same element of "giving your enemy maximal possible slack" spirit.

Here, to establish the maximum and the threshold, a little bit math is involved. I will spare you any formalism, because the actual idea is very simple, some would say trivial.

Let us start with a special case. Say, we want to find documents that contain all n user query terms, what are those documents?

They must be those documents that contain the rarest term in the user query. Right? Regardless of how rare a term is, a document must have it to meet our requirement of containing all query terms. In fact, the document must contain any one of the given query terms. In other words, it is necessary and sufficient to use any one term's set of containing documents as the candidates, and we can then proceed to check these candidates to see if they also contain other query terms. Since picking any one term would be the same, we will choose the rarest term, for its containing document list is the shortest. All other documents can be safely skipped. Magic, yes?

Generalizing this special case, to find documents containing n-1 query terms, we only need to pick as candidates those documents containing the rarest OR the second rarest term, i.e. the union of them. All the rest of the documents can be safely ignored. Isn't this nice?

We can continue this all the way down to needing only one query term in the document. At this point, every document containing any query term becomes a potential candidate. This reverts back to the difficult problem of searching all matching documents, which WAND solves by skipping documents that are not going to make into top-K.

But we can do much better. Because the above mathematical property also allows us to skip documents that are not going to have the required number of overlaps with the query.

Say we are at a tier that requires t overlaps between a document and a query. We are walking a document candidate down the term rows, trying to find out how many terms this candidate hits. If this candidate has already hit h terms so far, and is now on row k. If we give this candidate maximal slack, meaning that we assume he will hit all the remaining terms, his hypothetical maximal hits would be h + (n - k - 1). If this number is less than t, he can be safely kicked out, because he will never make into the exclusive t club.

This new pruning condition is the main novelty of T-Wand in term of algorithm, as it has not been reported in the literature for search algorithms, as far as I can tell. With this pruning condition and candidate pre-filtering, we can already achieve about 65% overall speed of Lucene. Adding the pruning condition of WAND, we have the current T-Wand performance, which beats Lucene with ease.

Implementation Takeaways

I said beating Lucene with ease, because we did not do any extreme optimizations that are abundant in Lucene. Our code is idiomatic Clojure, with some additional help of Java data structures at critical performance hotspots. The code weights less than 600 lines, including comments.

Clojure

So why use Clojure? The short answer is that it makes programming fun.

It is fun to explore ideas with the language. One little known fact that I found about using Clojure, is that it is an excellent language for implementing published algorithms in academic papers. Most algorithmic papers include pseudo code, written in an imperative programming style. They seem to be a far cry from any functional programming code one would write.

However, I found the opposite to be true. It is easier to translate published imperative pseudo code into functional programming real code, than trying to turn imperative pseudo code into imperative real code. Looking at any code repository accompanying a paper, the imperative programming code in the repository bears no resemblance to the nice pseudo code shown in the paper.

With a functional programming language, on the other hand, the resulting real code will have a one-to-one correspondence with the pseudo code. If you know Clojure, you can look at my code of T-Wand, see the similarity between my function score-docs and the pseudo code in [4]. I just gave each code block in the pseudo code a name and turns that into a function, find-pivot, score-pivot, next-candidates, etc. The function realizes the intention and logic of the pseudo code, but not following its style. The code is easy to follow and change, allowing exploration of many ideas.

The current form of T-Wand is the result of at least three major iterations of ideas, first the candidate pruning by required overlaps idea, then candidate pre-filtering idea, finally incorporated both into T-Wand. All these changes and refactoring were done with great ease, with a sense of adventure and fun. I can not imagine how painful this would be to iterate these ideas on a statically typed language and in an imperative programming style. I probably would just give up.

Index Storage

In Datalevin search engine, we store the primary data structure for searching in LMDB in an aggregated form, as a sparse integer list. The reason to store and access term information in aggregates, instead of storing them in inverted lists and accessed piece-meal, is because LMDB is very efficient (more efficient than file systems) at reading/writing large binary blobs.

As long as the data structure to be stored has high (de)serialization speed, it is more efficient to store/retrieve them in large aggregates than in more granular forms. This does not apply to more complex data structures with high construction cost, such as hash maps. We consider these as general advises on how to best use the so called single-level storage, such as LMDB. We earned these lessons in our many iterations of trying to find the best performing indexing data structure.

Conclusion

I am happy that this exploration turns out well. I would be thrilled to see T-Wand algorithm makes into other search engines, as it does improve user experience, performs well, and it is simple to implement in existing code bases that use WAND. There are many more ideas of improvement can be further explored, I would be happy to have a collaboration if someone wants to take it further.

Updates for Hacker News People

For people who are worried, please be assured that T-Wand also uses tf-idf and the vector space model, so you are not missing anything fundamental.

For people who say "Lucene is not properly used", please write the proper Lucene code in one of the issues, in Java if you want, I can integrate them into the benchmark. If you know Clojure, just send a PR. I appreciate that. Better yet, if you know Lucene so well, you should send a PR to Lucene that integrates T-Wand into Lucene, that would make the biggest impact.

For people who say "Does yours have all the Lucene features - stemming, incremental index, phrase search, range search, arbitrary sortable fields, etc?", at the moment, the following features are implemented:

Core search capability
English analyzer
Incremental and bulk indexing

The following are to be implemented in the future:

Fuzzy query matching
Stemming
Phrase search
Boolean operators

The followings are not necessary because Datalevin is a real database and it has more powerful database features than whatever a search engine can offer:

Range query is a foundation of Datalevin database
Fields are just watered-down columns in a database. Datalevin is a fully featured database. Whatever you want to do with fields, you can do with Datalog.

I am opening up for feature suggestions. Please file issues or send PR. I appreciate them.

Finally, please do not misunderstand what is going on here: I am running a startup, and I am also old enough to not care about publications as much as much as people who are younger or in academia. That's why I chose to reveal this in a blog post instead of hiding it until after my paper is published. A blog post can reach more people then an academic paper can.

References

[1] Broder, Carmel, Herscovici, Soffer and Zien, Efficient Query Evaluation using a Two-Level Retrieval Process, CIKM'2003.

[2] Ding and Suel. Faster top-k document retrieval using block-max indexes. SIGIR'2011.

[3] Okazaki and Tsujii, Simple and Efficient Algorithm for Approximate Dictionary Matching. COLING '2010.

[4] Crane, et al. A comparison of document-at-a-time and score-at-a-time query evaluation. WSDM' 2017.

Sync time for FreeIPA hosts

2021-04-08T07:41:14+00:00

Recently, I upgraded the FreeIPA server for my network to the latest version (4.8.10). Some strange things related to authentication started to happen with some services on my network.

For our Web site, netlify gotrue is used to enable users to login using SAML Single Sign-On. This started to fail due to this error:

SAML response has invalid time

For our discourse based forum, users started to experience failure to login using Google Single Sign-On. Again, it is time related:

Unable to verify authorization token due to server clock differences. Please try again.

So I check the time on the servers, they all have a few seconds differences from each other.

All my nodes use chronyd to manage NTP service. When run

chronyc sources -v

I found that all of the NTP client show something like this:

210 Number of sources = 1

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? ipa.example.com         0   8     0     -     +0ns[   +0ns] +/-    0ns

It means that the NTP server is not reachable. No wonder the times are out of sync.

So I checked on my IPA server, it turns out that the new version of FreeIPA server default to disable the NTP server and does not allow external NTP clients to access. The NTP port 123 is not open:

ss -lnp | grep "123"

This shows nothing.

To fix, edit /etc/chrony/chrony.conf, and add

allow all

local stratum 10

Then systemctl restart chronyd to restart the server.

On the client nodes, do the same, now the time should be synced.

210 Number of sources = 1

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ipa.example.com         4   7   377    73   +360us[ +447us] +/-   28ms

All those pesky authentication problems went away.

Dear Taiwanese friends, here is how to gain independence from China

2021-03-31T18:23:22+00:00

Is it possible for Taiwan to gain independence from China?

The answer is, yes. Here's how.

Before we begin, let's set aside the silly notion that "Taiwan is already independent". Statehood, like marriage, is something more about the formality than substance. You are either an independent country, or you are not.

Yes, you have an army, your own currency, your own passport, your elections, your government, so on and so forth. But, so do most of the large scale armed insurgences.

If you are serious about Taiwan independence, you will have to come to term with the formality. That is to say, to obtain a formal recognization of your statehood from the international community. Particularly when your statehood is contested by a big country from which you try to depart, that country's consent is pretty much a requirement. Remember, British agreed to USA's independence in the end. You, too, will have to obtain China's consent.

How is that even possible, you may ask? Well, that's what I am about to tell you. Read on.

First, let's look at the big picture. Do you know how many neighboring countries China has today?

The short answer is "a lot". The long answer is the following: on land, there are 14 countries: Afghanistan, Bhutan, India, Kazakhstan, North Korea, Kyrgyzstan, Laos, Mongolia, Myanmar, Nepal, Pakistan, Russia, Tajikistan, and Vietnam; on the sea, there are 6 countries: Brunei, Indonesia, Japan, South Korea, Malaysia, and Philippines.

In total, twenty countries are in peaceful coexistence with China side by side! As you can also see, many of them are tiny countries that you may not even know about. Clearly, that "China bad" narrative you are made to believe does not hold much water.

With so many neighboring countries, is it really a big deal that Taiwan is added to that list?

Rationally, not really.

Seriously? You ask. Yes, seriously.

Let me put it this way: in that list of China's neighbors, there is a country that used to be a Chinese territory for more than one thousand years! It is unbelievable, right? But it is true.

If you do not know, that country is Vietnam.

China has no problem with Vietnam being a separate country, which shares a land border with China, you think China will have a problem with Taiwan being as a separate country, which is located hundreds of miles away in Pacific?

So why is Taiwan special? What is the deal with Chinese demanding everyone else to recognize Taiwan as a part of China? What's the difference?

The answer will be counter-intuitive, but it makes sense if you think about it. Basically, it is less about what China thinks, but more to do with how Taiwan acts.

The gist of it, is that Taiwan does not act like a neighboring country of China, instead, Taiwan acts like a renegade Chinese insurgency, that's why China has to deal with her as such.

Taiwan acts like a Chinese insurgency

Let's postpone the discussion of history for the moment, just focus on the current affairs. Let us explore Taiwan's behaviors, shall we?

Taiwan government funds anti-Chinese government propaganda

First, it is well documented that Taiwan government is one of the primary funding sources of anti-Chinese government propaganda around the world.

We will not go into the details of Taiwan's funding of Chinese dissidents, such as Wang Dan, who lived in Taiwan for many years. If you are reading this, you probably have also heard about the so called "1450 net army", which is funded by the Taiwan government to wage propaganda war against any Chinese interest on the Internet.

Let's just look at Taiwan influence in USA, for example. Here is a detailed report on Taiwan funding of think tanks in Washington:

The five think tanks — the Brookings Institution, the Center for American Progress, the Center for a New American Security, the Center for Strategic and International Studies, and the Hudson Institute — all disclose their funding from TECRO but bury it deep on their websites or annual reports.

TECRO is the acronym for the Taipei Economic and Cultural Representative Office, which is an office under the Ministry of Foreign Affairs of Republic of China (i.e. the Taiwan government).

As a US citizen, I have to say that I find this foreign interference in our policy making very troubling. For the main thesis of this article, it is worth pointing out that these think tanks often propose policies that antagonize China and promote military confrontation against her. For example, this recent policy proposal talks about "guerilla sea denial", "guerilla air denial", and even “deterrence by punishment", which is defined as "Instead of threatening China’s invading forces, these capabilities would seek to deter China by threatening countervalue targets with conventional weapons."

So let me get this straight, the Taiwan government uses the tax money you paid to give to people that promote fighting a hot war on your island, so that they can profit from selling arms to you. Is that really to your best interest?

As we can see, Taiwan government is basically act as an enemy of the Chinese government. Perhaps it is because, officially, there is still a state of war between them. There is not even a cease-fire declared between the two parties that have engaged in a civil war since 1946. No truce, no cease-fire. That's the reality.

Taiwan government still claims ownership of the whole China

Officially, Taiwan government is calling herself the Republic of China, and her constitution still claims the ownership of the whole of China. Sorry, I am mistaken, she actually claims more than what the Chinese government claims. For example, the Republic of China does not recognize Mongolia as a separate country, whereas the People's Republic of China does.

This is not funny or it's some kind of schizophrenia or psychosis. The Taiwan government's claim actually has substance.

Taiwan holds Chinese treasury

Before the ruling party of Republic of China, the KMT, evacuated to the island of Taiwan, she took the whole of Chinese treasury with her. That included all the gold reserve of China, and more importantly, all of the cultural treasures of the Chinese civilization that could be moved.

Now, the Chinese civilization boasts a 5000 years of history and culture. Some would say the official religion of the Chinese civilization is not Confucianism, Taoism, Buddhism or any of the actual religions, but a belief in her own heritage and culture. In fact, Taiwanese often boasts of their better inheritance of the Chinese culture, since they still use traditional Chinese script, not the simplified one used in the mainland China.

Now you have it, both sides of the Taiwan strait see each other as a renegade insurgency that tries to usurp the power to govern China.

Therefore, Chinese government has the full right to pressure any other countries that hope to have a normal relation with her to recognize her sovereignty over the whole of China, rather than that of the Taiwan government's, just like Taiwan government did so before they were booted from the United Nations in 1970s.

Steps to earn Taiwan independence from China

With all the above established, for a rational person, it should be easy to see how to gain independence from China. The steps required are not complicated and easy to implement. Most importantly, it has been done before, by many people. Some of them were even successful.

Cease hostility against the Chinese government

This is a no-brainer. If you want to be a neighboring country of China, act like one. Unless you want to fight a war with your neighbor, you don't actively fund the forces to sabotage her interests wherever you can, right?

You can start by stopping writing the checks to all the anti-China individuals or institutions. It is so easy to not write the checks. Am I right?

Vietnam earned her independence from China, by agreeing to some concessions, not being act like an enemy of the Chinese government. In the end of the day, it is her consent that you are seeking.

Return Chinese cultural treasures

As an additional show of good will, you should offer to return all the cultural treasures that the previous Chinese government brought to your island as they fled from the country. These are Chinese things. You don't want to have anything to do with the Chinese, remember?

You may say, how about the British, the Americans, etc. why don't you ask them to return the Chinese treasures that they looted. Well, they looted. They fought the Chinese government at that time and won, so they took some things they liked. Did you fight and win against the Chinese government? No, you did not. So what right do you have to keep the treasures?

Abolish Chinese script

First, is to forge an independent sense of identity. You are well on the way there. For example, you insist that people calling you Taiwanese, not Chinese. That's good, but it is far from enough.

You really need to stop using Chinese, if you are truly serious. Invent a script of your own, like how Vietnam and Korea did it. Otherwise, it is only a matter of time, before you are absorbed into the sinosphere culturally and your own identity dissipates.

Unlike other cultures, the Chinese culture prizes unity over everything else, so you are not safe! Have you not noticed many of your celebrities are making a living in mainland China? That's dangerous.

You want to insulate your people from Chinese culture. Using traditional Chinese is not enough of an insulation. If someone can read simplified Chinese, they can read traditional Chinese as well, and can type traditional Chinese on a computer. There is also a resurgence of use of traditional Chinese in the mainland. It is only a matter of time, before you are fully absorbed.

Again, it is just too dangerous to continue using Chinese. Invent a script and mandate its use. It has proven to work, like it did in Vietnam, Korean, or more controversially, Japan.

Get ready to fight on your own

Now comes the hardest step. Once you declare independence, there will be a war. That is inevitable. No question about it.

However, a war does not mean defeat. You can win.

I mean, not "win" win, but achieving independence in the end. All you need to have, is a will to fight, on you own.

No, don't even think about it. I know what you are thinking. You think USA and her allies will come to you aid when the war with China broke out. I don't want to argue that USA and her allies will not, but I want to argue from another point of view, that is the pride of a nation.

Regardless what USA and her allies will do, you must be ready to fight like you want your independence and you are a nation of your own. To put it more bluntly, you have to prove your nationhood with the blood of the young.

You need to fight hard and drag the war on, so that the cost of keeping the war going outweighs the perceived benefits from your adversary's point of view. That's your goal.

That's how Vietnam got her independence. She fought, won some battles, lost some too. But she kept fighting, until the Chinese emperor got tired of fighting with her. When the negotiation offer came, she took it. Obviously, she had to make some concessions. But she got her independence in the end.

Conclusion

There you have it, a simple recipe to achieve independence from China. It has proven to work, and it is not complicated.

Stop hostility against China
Return Chinese treasures
Abolish Chinese Script
Prepare to fight for your nation's independence

What are you waiting for, go for it. Put pressure on your representatives to implement these steps. You are a democracy, ain't you?

How much can a Clojure developer do alone?

2021-03-23T17:45:35+00:00

Someone asked on Reddit this question, for he's wondering if it is worth the time to learn Clojure well. He has dabbled in Clojure for half a year, but was not convinced of its benefits. He has not used REPL much, and was wondering if he was doing something wrong. He additionally has some related questions about the Clojure and functional programming in general, here are my two cents of an answer.

I am not using the REPL much, am I doing something wrong?

Yes, if you are not using the REPL, you are not doing Clojure right. That's the answer. It may not be what one likes to hear, but it is true.

There are already a lot of blog posts and Youtube videos about what it means to be "using the REPL" in Clojure, or in Lisp in general. None of them articulated the points from a Human-computer Interaction point of view. As someone who earned a Ph.D. in that field, I think I am qualified to make this point.

First, "using the REPL" does NOT mean typing your code in REPL. Heck, no, it's the opposite.

"Using the REPL" actually means typing your code in your favorite editor. However, one sends the code to evaluate in a REPL, which may not even be visible at all. Once the code is evaluated, the results immediately show up in the editor, so you get the feedback right away. Obviously, this requires some upfront setup, but a proper Clojure REPL setup exists in all major editors. Just google it.

Note that one normally sends the code to evaluate with a single key stroke. Note also, a very important point that people often miss, is that this "sending code to evaluate" is uniquely convenient in Lisp because of the parentheses.

There's a notion of "form" in Lisp, that is the code enclosed between a pair of parentheses, which can be independently evaluated. So, when next time someone insists that their favorite non-Lisp language also has a REPL, ask them, does it have a notion of "form"?

The benefit of a form, is that one no longer needs to use a mouse or some awkward key combinations to painstakingly select a region of code first, before sending it out for evaluation. Instead, one can use a single key stroke that means "evaluate the form under the cursor", or "evaluate the form before the cursor", etc, to precisely define the scope and send the code at the same time.

As someone who has published papers in human motor controls, I can assure you this simplification makes a huge difference in ergonomics. It reduces the cost of using a REPL significantly, to the extent, I would venture to estimate that half of the potential productivity gains (if any) of the Lisp family of languages come from using the REPL this way.

Psychologically, using REPL this way improves the flow, the complete immersion in one's activity without interruption. No wonder Clojure programmers are found to the happiest: a mental state of flow improves well-being and life satisfaction.

Please, do yourself a favor, go learn and practice the skills of REPL-driven programming in Clojure. There are many videos that show how it is done.

Can I learn Clojure well all by myself?

For this one, my opinion is "probably no".

As alluded to above, significant amount of "doing Clojure right" is about some tiny bits of seemly unimportant things, but they add up. That's why I hold this unpopular opinion among Clojurians that Clojure is not a particularly good language for self-learning, unless one is very good at self-starting, which is not the case for the vast majority of people.

Most people need some trainings to be able to get into something new. If they have not been taught REPL driven programming, I don't think they can discover it by themselves, despite maybe hearing others talking about it constantly. Heck, if left alone, there are people who would complain about having to count parentheses, instead of taking the initiative to find tools to help themselves, for which there are plenty of options, e.g. paredit, smartparens, parinfer, and so on.

On the other hand, I seldom had any problems turning my fresh college graduates new hires into competent Clojure programmers within less than a month. The differences are two fold:

Pride. Fresh college graduates have none. When they encountered obstacles, they just ask questions or try harder, instead of going to the easier route of blaming the technology. Clojure is a particularly easy target to blame: It's niche, a Lisp, a JVM language, dynamic typing, etc, the list can go on and on.

Training. I made sure that my fresh college graduates were trained, down to the development environment setup. They also start working on a good code base right away. So they pick up the right habits at the beginning and never have to experience the kind of detours a self learner has to go through. There are lots of habits and setups that experienced Clojurians take for granted, but one has to see them to even know about them.

So what's the solution? I don't know. The only thing I can think of, is hoping that more Clojure shops are successful so we can hire more people, or Clojure finds a niche that it occupies fully so that when the time comes for that niche to explode, Clojure grows accordingly.

For whoever is reading this though, the situation is much better, because you are already reaching out and are not trying to learn all by yourself. The Clojure community is one of the nicest programming language communities. People are very welcoming to newcomers and are very helpful. Please reach out and get help. Here are some sites to get you started:

How much can a Clojure developer do alone?

Now back to the main question, how productive can a Clojure programmer become? There are also a few related questions, I will answer one by one.

Does it really make such a difference to use Clojure?

Yes, it really makes a huge difference to use Clojure. Think of it this way, Clojure is a language designed for software consultants by software consultants. Cognitect, the company behind Clojure, was a software consultancy, now acquired by a bank. Clojure is also ideal for a startup, where a couple of competent Clojure programmers can write a complex application that would took a huge team of developers in other languages years of work. For my startup, our clients always thought we had a 30+ person engineering team when we have had never more than half a dozen developers.

How much faster does it made you develop software? And in which areas? Did it improve performance because of easier composability? Did it made the code easier to understand because of the "better" abstractions once you got firm with them?"

All of the above. Composability, easier to understand, better abstraction, etc.

It is much faster to develop in Clojure. The language creator, Rich Hickey, made all the design choices to ensure that a software consultant can deliver good software in as short as possible time frame, so that this software consultant can make the most profit. For a software consultant, time is the main cost, so time to market is optimized in Clojure.

Technically, Clojure adds to Lisp the novel ingredients of immutable data, which significantly simplifies software development. There is no other language that places such an emphasis on programming directly with plain and naked data literals. Think of programming as working with a richer JSON format without all that serializations and transformations. How nice would that be? Well, that's exactly what Clojure programming is like.

Another important point is that Clojure is a very concise language. Empirical data shows that one of the only things reliably correlated with software quality is the number of lines of code. The more lines of code, the more bugs there will be. Clojure is one of the only four languages where language choices is statistically significantly correlated with software quality in this large scale study. The other two "good" languages are found to be Haskell and Ruby, the one "bad" language is C++, while Clojure has the highest statistical significance number on the "good" side.

Could you possibly develop a complex enterprise software in Clojure with just say 4-8 people, knowing Clojure really well?

Of course. It is not just possible, but it is also optimized for it. Clojure was developed to write complex enterprise software where user requirements are arbitrary and constantly changing. Please watch any of the many Rich Hickey's videos, as he articulated the motivation well. Plenty of big enterprises are taking advantage of Clojure, e.g. Amazon, Walmart, and so on.

Personally, knowing Clojure let me dare to do more ambitious things. For example, I wouldn't have started my company if I had not known Clojure. Before we had employees, I alone wrote the core of our chatbot platform, including a compiled domain specific language and its runtime, plus the Web front end and backend. With a small team, we had written and re-written the business facing Web application several times, which includes a no-code designer UI and an online IDE. We also have an API for programmers. All these development are done in Clojure, except some machine learning pieces done in Python.

Here's my concern: I felt like, once you got functions on which other functions also depend on and expect a certain signature, you would also get a tree-like complexity where you have to scratch your head for a while to figure out how to solve it. And if the code base gets large enough, I can imagine that understanding the code and all the context just gets equally complex.

The real situation in Clojure is actually a lot better than your imagination, because the immutable data localizes all functionalities, so one never has to look at the whole dependency graph to understand the code or to solve a problem. This is not something that can be said for Java-like object oriented programming, where functionalities are spread out into many layers of abstractions and into many objects.

A properly structured Clojure code base uses a few namespaces of pure functions. I tend to avoid many tiny files with only a few functions in them. We are not writing Java, after all. Instead, group related functions into a large file, so that one seldom needs to look over multiple files to change something.

Again, I stress that in Clojure, because most functions are pure and dealing with immutable data, one does not need to keep all the complexity of the whole application in the head in order to solve problems. One mostly works locally.

In my opinion, Clojure programming is different only in the small. Programming in the large, like software architecture, is mostly similar to other languages. The only thing of note, is that one may place more focus on pure data when designing the APIs see e.g. my diff oriented architecture talk.

Finally, let me reveal a little publicized secret of programming. To navigate large code base, in addition to the usual "go to definition" hot keys in your IDEs or editors, a good programmer, in any language, relies heavily on text search, even in their own code bases. This is a skill that experienced coders tend not to talk publicly a lot about. So please know the searchers of your editor very very well, be it grep, silver, platinum, or whatever.

What are the key skills a Clojure developer has to develop to gain from that language?

Having humility and ambition at the same time. The humility to know there is a lot to learn and the ambition to dare to do things that most thought impossible.

"Writing C code in Java/Clojure: GraalVM specific programming"

2021-02-08T05:41:23+00:00

One of the latest fashions in the Java world is GraalVM. For someone who has been around, I still remember the "Write once, run anywhere" slogan of Java virtual machine. Apparently, the wheel has spun back, now people want to write native code in Java, which has to be compiled to different machine codes in different platforms.

The driving forces for this change may include the often lamented slow startup time of Java programs. For the brave new world of serverless functions, a quick starting program makes a lot of economic sense. The small but vibrant Clojure community, which I consider myself a part of, is particularly excited about this new found capability of JVM, for we now have one more way to write quick starting command line programs using our beloved language. The other way is to use ClojureScript on one of the Javascript engines, but Javascript engines are slower than JVM and are not as nice. For example, Babashka is one such example that has taken the community by storm.

As the author of Datalevin, a relatively new open source Datalog database, I have decided to answer the users' call for a native version of Datalevin using GraalVM native image technology. After some trials and errors, I have gotten a native version of Datalevin to pass all the tests. Now I can share some experience.

Failed attempts to compile Datalevn to native image

The difficulty of directly compiling Datalevin to GraalVM native image lies in our use of JNR library to wrap LMDB, the underlying key-value store that is written in C. As far as I know, no one has succeeded in getting JNR to work with native image. This is not for the lack of trying, see for example this effort and this issue. I myself have tried really hard. Although I think I have gone further than many, the need to rerun some class initialization at runtime prevented me from succeeding. Apparently, some GraalVM team members also think it is not easy to do at this point.

I am left with two options: one is to use a different LMDB wrapper that does not use JNR. For example, a JNI based LMDB wrapper is available and GraalVM native image does support JNI. However, that code has not been updated for a long time and its maintainer seems to have joined LMDBJava, the LMDB wrapper that uses JNR. The other option is to write a LMDB wrapper of my own that can run in native image. So I did just that.

Writing C code in Java/Clojure

As a native technology, GraalVM obviously has the facility to interface with C code. Not just that, it must also have the facility to write native code on its own. Fortunately, such facility is also packaged as a SDK that is publicly available. To use this SDK, one has to be familiar with not just Java, but also C programming, because effectively, it is writing low level C code in Java syntax.

The only problem is that there isn't an official documentation for this kind of programming, just a Java doc and a coding example. After some research, I also found a few other examples: an OpenGL demo, a neo4j native driver and an uname utility. I hope that this blog post adds to this growing library of GraalVM specific programming examples.

Import C data and functions

Our goal is to write a LMDB wrapper with GraalVM SDK without using JNI.

The first step is to make Java aware of the LMDB C data structures and functions declared in the header file lmdb.h, so that our Java code can use them. This step is fairly easy and pleasant, compared with JNI. All one needs to do is to write a Java class to list all those C structs, enums and functions as native interfaces and annotate these with appropriate GraalVM specific annotations.

For example, for a C struct definition:

typedef struct MDB_val {
    size_t       mv_size;   /**< size of the data item */
    void        *mv_data;   /**< address of the data item */
} MDB_val;

The corresponding Java declaration is the following:

    @CStruct("MDB_val")
    public interface MDB_val extends PointerBase {

        @CField("mv_size")
        long get_mv_size();

        @CField("mv_size")
        void set_mv_size(long value);

        @CField("mv_data")
        VoidPointer get_mv_data();

        @CField("mv_data")
        void set_mv_data(VoidPointer value);
    }

The CStruct annotation tells Java which C struct to import. I gave the Java interface the same name as C so it is easier for me to keep track. One could also name it otherwise, e.g. "MDBVal" if one wants to follow CamelCases. It does not matter.

PointerBase interface indicates a native word type, and is the root of all C pointers being imported into Java. Most things in GraalVM SDK work with these native word types.

CField annotation imports a C struct field as a Java method. Setters and getters need to be declared separately. Again, I give these methods similar names as C field names, but you may prefer to follow Java name convention instead, e.g. "getMvSize".

VoidPointer is a faithful translation of void *.

For an opaque C struct declaration that does not list its fields, one must add an isIncomplete option to the annotation, otherwise compilation will fail with a "sizeOf" related error. For example, for a C struct like this:

typedef struct MDB_env MDB_env;

The corresponding Java declaration is this:

    @CStruct(value = "MDB_env", isIncomplete = true)
    public interface MDB_env extends PointerBase {}

C functions are directly translated into Java static native methods, e.g.:

int  mdb_env_get_maxkeysize(MDB_env *env);

The corresponding Java declaration is the following:

    @CFunction("mdb_env_get_maxkeysize")
    public static native int mdb_env_get_maxkeysize(MDB_env env);

Notice that the pointer to MDB_env struct is replaced by MDB_env Java interface declared above, as it extends PointerBase.

For parameters with double pointer type, one can declare them as WordPointer in Java, as they point to another pointer (i.e. word). For example, this C function:

int  mdb_env_create(MDB_env **env);

is translated into this Java code:

    @CFunction("mdb_env_create")
    public static native int mdb_env_create(WordPointer envPtr);

Finally, for these annotations to work, the containing class must be annotated with appropriate C context, which is often defined as a static inner class of the class, like so:

@CContext(Lib.Directives.class)
public final class Lib {

    public static final class Directives implements CContext.Directives {
        @Override
        public List<String> getHeaderFiles() {
            return Collections.singletonList("<lmdb.h>");
        }

        @Override
        public List<String> getLibraries() {
            return Arrays.asList("lmdb");
        }
    }
    ...
}

The CContext.Directives class imports the necessary C header file and finds the corresponding C library. In this particular example, we are importing the system wide LMDB header file and library installed by apt install liblmdb-dev on Debian/Ubuntu. An example of importing local header file and library can be found in Datalevin source code here.

The code of all C to Java translations is here

Clojure specific considerations

Clojure deftype supports Java annotations, so that is what I used to implement the higher level constructs of the LMDB wrapper. For example:

(deftype ^{Retention RetentionPolicy/RUNTIME
           CContext  {:value Lib$Directives}}
    LMDB [^Env env
          ^String dir
          ^RtxPool pool
          ^ConcurrentHashMap dbis
          ^:volatile-mutable closed?]
  ...)

One limitation that one needs to be aware of when writing native image related Clojure code, is that most things in the GraalVM SDK inherit from org.graalvm.word.WordBase, not from java.lang.Object, which breaks the hidden assumption of a lot of Clojure constructs. For example, one cannot do this:

(let [env ^Lib$MDB_env (Env/create)]
  ...)

Because Lib$MDB_env extends PointerBase, but Clojure let seems to assume everything getting bound is an Object. An "expecting Object but got Word" or vice visa error will be thrown during compilation for these cases. So writing a thin layer of wrapper for these GraalVM word types seems to be inevitable, unless Clojure can be enhanced to be more native image programming friendly.

Another point of caution is about arrays. GraalVM uses reflection to create array objects, but if these array objects are not declared at build time, the code won't run. For instance, for Clojure into-array function, we should not omit the optional first argument for element data type, otherwise, one has to manually specify the array type in the reflection.json file during compilation, or the Graal runtime will complain about "Class such and such is instantiated reflectively but was never registered". For Clojure dynamically generated classes with names such as these, "datalevin.test.query$fn__12734$fn__12739$fn__12740[]", it is impossible to add them manually in reflection.json. So the only solution is to specify the element data type in code, e.g. (into-array Object data), instead of (into-array data).

The full Clojure project for native Datalevin is here.

Memory management and pointer wrangling

The main challenge of building a LMDB wrapper is to find a way to put transaction data into and get query data out of LMDB. As shown above, LMDB use a MDB_val struct to represents input/output data. All it contains is a data size and a pointer to the data. LMDBJava uses JNR and Unsafe or reflections to manipulate a java.nio.ByteBuffer to achieve this. Since we cannot use these in this project, we have to come up with a GraalVM specific solution.

It turned out the code to do this is quite easy to write. Instead of allocating the ByteBuffer in Java and presenting it to C, we manage the memory in C and present it as a ByteBuffer in Java, without all that Unsafe and reflection shenanigans.

/**
 * Wrap LMDB MDB_val to look like a ByteBuffer at the Java side
 */
@CContext(Lib.Directives.class)
public class BufVal {

    private ByteBuffer inBuf;

    private VoidPointer data;
    private Lib.MDB_val ptr;

    public BufVal(int size) {

        data = UnmanagedMemory.calloc(size);
        ptr = UnmanagedMemory.calloc(SizeOf.get(Lib.MDB_val.class));

        ptr.set_mv_size(size);
        ptr.set_mv_data(data);

        inBuf = CTypeConversion.asByteBuffer(data, size);
        inBuf.order(ByteOrder.BIG_ENDIAN);
    }

    /**
     * Set the limit of internal ByteBuffer to the current position, and update
     * the MDB_val size to be the same, so no unnecessary bytes are written
     */
    public void flip() {
        inBuf.flip();
        ptr.set_mv_size(inBuf.limit());
    }

    /**
     * Set the limit of internal ByteBuffer to capacity, and update
     * the MDB_val size to be the same, so it is ready to accept writes
     */
    public void clear() {
        inBuf.clear();
        ptr.set_mv_size(inBuf.limit());
    }

    /**
     * Free memory
     */
    public void close() {
        UnmanagedMemory.free(data);
        UnmanagedMemory.free(ptr);
    }

    /**
     * Return a ByteBuffer for getting data out of MDB_val
     */
    public ByteBuffer outBuf() {
        ByteBuffer buf = CTypeConversion.asByteBuffer(ptr.get_mv_data(),
                                                      (int)ptr.get_mv_size());
        buf.order(ByteOrder.BIG_ENDIAN);
        return buf;
    }

    /**
     * Reset MDB_val pointer back to internal ByteBuffer, and return it
     * for putting data into MDB_val
     */
    public ByteBuffer inBuf() {
        ptr.set_mv_data(data);
        return inBuf;
    }

    /**
     * Return the MDB_val pointer to be used in LMDB calls
     */
    public Lib.MDB_val getVal() {
        return (Lib.MDB_val)ptr;
    }

    /**
     * factory method to create an instance
     */
    public static BufVal create(int size) {
        return new BufVal(size);
    }
}

We allocate the memory for the data and the MDB_val struct with the UnmanagedMemory.calloc static method from the SDK. This allocates memory from the heap just like in C. We will then need to free the memory ourselves.

If the memory is needed only for a short period of time, the other options are PinnedObject or StackValue classes of the SDK. The former allows creating Java objects and then pinning them down, so that the garbage collector does not move them or delete them, in order to get a stable pointer to work with at the C side. The latter allocates objects from the stack so it is short-lived. These cases do not fit our need for long term pointers to database data structures, so we use UnmanagedMemory.

The SDK utility CTypeConversion.asByteBuffer static method is what makes our effort possible. We can simply expose a MDB_val as a ByteBuffer to Java in the constructor for incoming data, and in outBuf() for outgoing data. The rest of the code is just bookkeeping for the ByteBuffer.

Conclusion

I am happy that this effort is turning out well. The GraalVM SDK is quite pleasant to use, once one figures it out. I wish that this API can be made available in regular JVM as well, so we can truly write it once, and use it everywhere, regardless the underlying languages and platforms.

Important Update (2023-01-06)

For GraalVM versions newer than 21.3.0, those C context annotations in the Clojure code shown above should be removed, because CContext is now marked as HOST_ONLY, and is available only during build time. Basically, only those Java classes that directly wrap the C code should now add these annotations.

Fix FreeIPA httpd Let's Encrypt Certificate Update

2021-01-01T22:58:21+00:00

For a public facing Web interface of FreeIPA server, it is desirable to use a 3rd party SSL certificate issued by a commonly accepted certificate authority, rather than using the server's own. Let's Encrypt provides free SSL certificate for this purpose.

To use Letsencrypt certificate with FreeIPA, this script does a good job, and I have been using this for several years.

Problem

When the letsencrypt certificate was renewed last month, a problem occurred. The Apache httpd server used by FreeIPA could not start. The error in /var/log/httpd/error_log says:

SSL Library Error: -8179 Certificate is signed by an unknown issuer

The next line says:

Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved.

So I did just that. Now the httpd server could start, and the Web UI did show up. However, one could not login, and the UI showed an "unknown error" message.

The error log says:

ipa: INFO: 401 Unauthorized: HTTPSConnectionPool(host='ipa.example.com', port=443): Max retries exceeded with url: /ipa/session/cookie (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:866)'),))

and

SSL Library Error: -12195 Peer does not recognize and trust the CA that issued your certificate

That's a bit frustrating.

What's worse, in my attempt to reran the "renew-le.sh" script to see what's going on, I ran into the rate limit of letsencrypt service, which only allows 5 renewals per week. There's no way to reset that and I had to wait for a week.

Fix

Today, a week has past, and I finally fixed the problem, after almost half a day.

It turned out that Let's Encrypt has been updating their Root and Intermediate certificates. See details here.

The old root certificate, DSTRootCAX3.pem is being phased out. In the past, I have been getting certificate from X3 under it. However, the new certificate that I got is from R3 under the new root isrgrootx1.pem.

What's worse is that this new certificate was still issued with old root. What's missing, is an intermediate certificate that crossed signed with this old root, lets-encrypt-r3-cross-signed.pem. Without this, the certificate cannot be verified, hence the error.

So the solution, is to download https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem, then install it:

wget https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem

ipa-cacert-manage install lets-encrypt-r3-cross-signed.pem -n letsencryptr3-cross -t C,,

After this, if you attempt to run ipa-certupdate, it will fail:

ipapython.admintool: DEBUG: The ipa-certupdate command failed, exception: NetworkError: cannot connect to 'https://ipa.example.com/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)

ipapython.admintool: ERROR: cannot connect to 'https://ipa.example.com/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)

ipapython.admintool: ERROR: The ipa-certupdate command failed.

It's like catch 22, because your httpd SSL certificate cannot be verified, you cannot update your ipa certificates, because it requires connecting to your httpd server over SSL, but it is already failed due to lack of proper certificate. What a stupid design!

Anyhow, I finally figured out a way around this, which is to manually edit /etc/ipa/ca.cert, and append all the certificates needed for the full chain: DSTRootCAX3.pem -> lets-encrypt-r3-cross-signed.pem -> cert.pem that was issued by letsencrypt, one after another.

Restart server. Now ipa-certupdate is successful! All is well.

Some useful commands

In the process of figuring this out, I found a few commands that could be useful in the future to diagnose similar problems.

To figure out what certificates FreeIPA knows about:

ldapsearch -Y GSSAPI -Q -b  cn=certificates,cn=ipa,cn=etc,dc=example,dc=com

To see what certificate httpd knows about:

certutil -L -d /etc/httpd/alias/

To install a certificate to httpd with name "Server-Cert":

certutil -A -d /etc/httpd/alias/ -n Server-Cert -t u,u,u -a -i file.pem

To test SSL connection:

openssl s_client -showcerts -verify 5 -connect ipa.example.com:443

ZGC garbage collector and Clojure applications

2020-12-29T22:20:27+00:00

The main product of my startup is written in Clojure, a language that puts enormous pressure on the memory garbage collector due to the pervasive use of immutable data structures. The new Z garbage collector on JVM has been a blessing for us, as it has largely solved our memory problems. However, there are a few points that one needs to pay attention.

Large pages

We followed the suggestion on the ZGC page to enable large pages.

Here is the easiest way to enable large page on Debian Linux: add "hugepages=9216" to GRUB_CMDLINE_LINUX_DEFAUT line in the /etc/default/grub file, issue update-grub as root, then reboot. This creates 9216 huge pages that are 2MB each (default), totaling 18GB. In the JVM startup options, add -XX:+UseLargePages to enable large pages in JVM.

The ZGC page says:

Configuring ZGC to use large pages will generally yield better performance (in terms of throughput, latency and start up time) and comes with no real disadvantage.

Unfortunately, that is not always true.

As mentioned, our Clojure application puts a lot of pressure on GC. After the application has been up for a few days and after heavy loads, it will become extremely slow, resulting in customer complaints. We had to restart the application when that happened.

Disabling large pages (reverse the above steps) removed the problem.

It appears that the culprit is memory fragmentation. As described here:

for a system that has been up for a long time, excessive fragmentation can make it impossible to reserve enough large page memory.

That is to say, be careful with large pages. Avoid it if memory pressure is high.

Number of Concurrent GC Threads

Since a Clojure application allocates tons of temporary objects then discards them, the garbage collector has more work to do. JVM's default setting of -XX:ConcGCThreads tends to be lower than necessary for a Clojure application under load. For example, by default, ZGC collector will only use one concurrent CG thread on a four core machine, which is definitely not enough. Setting the number to be two seems to be the minimum. I set it to three on our 8 core production servers. It seems to work well.

How to Setup SAML2 Authentication on Sentry with Keycloak

2020-10-19T18:28:25+00:00

With newer versions of on-premise Sentry (I am using Sentry 20.10.1), the LDAP authentication does not seem to work any more. The code still compiles but the LDAP login UI does not show up. Fortunately, newer versions of Sentry provide built-in support for SAML2 authentication, so we can use that instead. We can do this because our LDAP service is connected with an identity provider, in our case, a Keycloak server.

It took a bit of fiddlings in the UI of these two applications to set things up correctly. Here's how.

Keycloak Setup

The first step is to register Sentry with IdP, i.e. the Keycloak server. I am using Keycloak 3.4.3.Final community.

Create a client

Click "Clients" > "Create".

For the required "Client ID" field, just type something like https://sentry.example.com/saml/metadata/example/, assuming the sentry server is at sentry.example.com, and the organization slug in sentry is example.

Choose saml for "Client Protocol".

Click "Save".

Configure the client
"Sign Assertions" => OFF
"Encrypt Assertions" => OFF
"Client Signature Required" => OFF
"Force POST Binding" => ON
"Force Name ID Format" => OFF
"Name ID Format" => email
"Valid Redirect URIs" => *
"Assertion Consumer Service POST Binding URL" => https://sentry.example.com/saml/acs/example/
"Logout Service POST Binding URL" => https://sentry.example.com/saml/sls/example/

Leave the rest as they are.

Configure Mappers

Click "Delete" on the default "role list", and confirm, as we will use a builtin mapper.

Click "Add Builtin", check "X500 email", and click "Add selected".

Click "X500 Email", and change "SAML Attribute Name" to user_email, as that's what Sentry expects. Click Save.

We are done with Keycloak setup, now let's setup Sentry side.

Sentry Setup

The instruction on registering IdP with Sentry is pretty good.

The first method of "Using Metadata URL" works with Keycloak.

For "Meta URL", use https://idp.example.com/auth/realms/example.com/protocol/saml/descriptor, assuming the keycloak server is on idp.example.com, and the realm name in there is example.com.

For Attribute Mappings, use user_email for both "IdP User ID" and "User Email" required fields.

If everything setup correctly, after you are directed to your keycloak server to login, you should be directed back to Sentry with two green notifications on top, success!

LDAP Authentication Module for Nginx on Debian

2020-10-04T23:33:54+00:00

To control access to various internal Web sites of a company, a simple method is to enable LDAP authentication on the Web server, so that the company directory can be brought to bear and there is no need to create individual accounts for employees on different systems.

Nginx is one of the most popular free Web servers, it has a lot of built-in modules. Unfortunately, LDAP is not one of them, so we have to compile from the source.

Here are the steps to build a nginx Debian package from source with LDAP module enabled, on Debian 10 buster. While at it, we will also add a nginx virtual host traffic status module.

First install necessary tools:

sudo apt install dpkg-dev devscripts

Now get the source and dependency of nginx

sudo apt source nginx
sudo apt build-dep nginx

It is important to get the LDAP development library, as well as a few necessary libraries.

sudo apt install libldap2-dev libssl-dev libpcre3-dev

Clone nginx-auth-ldap and nginx-module-vts source code that we plan to compile into the nginx binary.

git clone https://github.com/kvspb/nginx-auth-ldap.git
git clone https://github.com/vozlt/nginx-module-vts.git

The part we need to change in the nginx source tree is the Debian rules, which govern how a Debian package is build. Goes into the debian directory of the nginx source folder, and edit the rules file:

cd nginx-1.14.2/debian
vi rules

Add two lines to the end of common_configure_flags

   ... \
   --add-module=<path to>/nginx-auth-ldap \
   --add-module=<path to>/nginx-module-vts

Now goes back and build.

cd ..
debuild -b -uc -us

This may take a while. If the build is successful, the parent directory will now contains a bunch of .deb files and these can be installed with dpkg -i. First install nginx-common_1.14.2-.deb, then libnginx-mod-.deb, finally, nginx-light, full, or extras.

How to Add a Preview Link to Self-hosted Netlify CMS

2020-04-09T21:47:15+00:00

If you are using a self-hosted Netlify CMS as the online editor for your SSG powered Website and you are using the editorial workflow (you have publish_mode: editorial_workflow in your config.yml), a pain point is that you do not see a preview of the live page when the page is in draft. Unlike the master branch that you can see the live page after it is built, the draft is committed to a different git branch so you do not have a link to the built page.

Sure, you always have a preview on the right panel in Netlify CMS, but this preview does not look like the real page. Even if you use CMS.registerPreviewStyle to register a stylesheet that matches your live page style, you still miss all the other parts, such the header, footer and sidebar, so you don't know how the whole page looks like until the page is published.

Netlify CMS has a feature to show a preview link for drafts, but one needs to do some setup to make it shows up. Right now, this feature only supports github as the git backend. Here I will show how I work with github, Jenkins and nginx to supply a preview link to Netlify CMS.

Before we begin, here's what we already have. We use a github organization, say example-com to host the repository for the site, so example-com/site would be the repo. The site source contains a Jenkinsfile, so when the repo is committed to, Jenkins automatically builds the site using its github integration.

We also set up nginx to directly point to the built site directory as the root. We are using eleventy as the SSG, so the built site is in _site directory. This works well for automatic deployment of the site. Whenever the master branch of the source repository is committed to, after a few seconds, the web server's <jenkins-agent-home>/workspace/<jenkins-job-name>_master/_site directory will contain the updated site content for https://example.com

To add a preview link, we basically need to do the same for the preview branch, PR-5, a pull-request to github sent by Netlify. There are a few things need to happen.

DNS

We need to setup a wildcard DNS record, so that HTTP requests to places such as PR-5.example.com, PR-11.example.com goes to the IP address of the Web server. Basically, you need to add an A record for *.example.com.

Nginx

On the Web server, we need to setup nginx to look for these PR host names, and dynamically set the site root to point to the correct Jenkins workspace directory. For example, PR-5.example.com should have <jenkins-agent-home>/workspace/<jenkins-job-name>_PR-5/_site as the root. To do that, first edit /etc/nginx/nginx.conf, add in the http context the following:

    map $http_host $rootpath {
        ~^(?<pr>pr\-.+)\.example\.com$  $pr;
    }

This will capture the PR-5 part of the hostname, and put it in $rootpath variable.

Then create a nginx site conf for the PR sites, say, /etc/nginx/site-enabled/pr.conf,

server {

    # SSL configuration
    #
    listen 443 ssl;
    listen [::]:443 ssl;

    ssl                  on;
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

        ssl_session_timeout  5m;
        ssl_ciphers  HIGH:!aNULL:!MD5;

    server_name ~^pr\-.+\.example\.com$;

    set_by_lua $pr "return string.upper(ngx.var.rootpath)";
    root /home/jenkins-slave/workspace/example-com_site_$pr/_site;

    location / {
        index index.html;
    }
}

Because DNS is case insensitive, it will not preserve the uppercase of PR-5.example.com, we will have to use lua to uppercase it to match the case sensitive directory path.

Notice also that we have a wildcard SSL certificate from Lets' Encrypt for *.example.com.

Jenkinsfile

On Jenkins, just setup a job that use a Jenkinsfile. My Jenkinsfile in the site source looks like this:

#!/usr/bin/env groovy

pipeline {
  agent { label 'web' }
  environment {
    GITHUB_ACCESS_TOKEN = credentials('GITHUB_ACCESS_TOKEN')
  }
  stages {
    stage('Build') {
      steps {
        sh "env"
        script {
          if (env.BRANCH_NAME.startsWith("PR-")) {
            env.BUILD_INFO = "<${env.RUN_DISPLAY_URL}|${env.JOB_NAME} [${env.BUILD_NUMBER}]> submitted by ${env.CHANGE_AUTHOR} with PR <https://github.com/example-com/site/pull/${CHANGE_ID}|#${env.CHANGE_ID}>: ${env.CHANGE_TITLE}"
          } else {
            env.GIT_COMMIT_MSG = sh (
              script: "git log --format=%B -n 1 ${env.GIT_COMMIT} | head -n 1",
              returnStdout: true).trim()
            env.GIT_AUTHOR_NAME = sh (
              script: "git show -s --pretty=%an ${env.GIT_COMMIT}",
              returnStdout: true).trim()
            env.BUILD_INFO = "<${env.RUN_DISPLAY_URL}|${env.JOB_NAME} [${env.BUILD_NUMBER}]> submitted by ${env.GIT_AUTHOR_NAME} with commit <https://github.com/example-com/site/commit/${env.GIT_COMMIT}|${env.GIT_COMMIT.take(7)}>: ${env.GIT_COMMIT_MSG}"
          }
        }
        sh '''
          npm install
          npx @11ty/eleventy
        '''
      }
    }
  }
  post {
    success {
      script {
        if (env.BRANCH_NAME.startsWith("PR-")) {
          sh '''
              GIT_PR_COMMIT=$(git show-ref -s "refs/remotes/origin/${BRANCH_NAME}")
              curl -X POST -H "Content-Type:application/json" \
              -H "Authorization: token ${GITHUB_ACCESS_TOKEN}" \
              -d '{"state": "success", "context": "netlify-cms/preview/deploy", "description": "Deploy preview ready", "target_url": "https://'"${BRANCH_NAME}"'.example.com/"}' \
              "https://api.github.com/repos/example-com/site/statuses/${GIT_PR_COMMIT}"
          '''
          }
      }
      slackSend (color: '#00FF00', message: "SUCCESSFUL: Job ${env.BUILD_INFO}")
    }
    aborted {
      slackSend (color: '#FF00FF', message: "ABORTED: Job ${env.BUILD_INFO}")
    }
    notBuilt {
      slackSend (color: '#AAAAAA', message: "NOT_BUILT: Job ${env.BUILD_INFO}")
    }
    unstable {
      slackSend (color: '#FFFF00', message: "UNSTABLE: Job ${env.BUILD_INFO}")
    }
  }
}

On github, I have created a personal access token, and added the token in Jenkins credentials as GITHUB_ACCESS_TOKEN.

You can see that we are sending build status to slack, but that's not important. The important part is in the additional step in the post success script, where we use curl to send a POST request to github statuses API to report a successful preview deployment (as a pull request). All the information needed for the preview link is in the JSON payload

{ 
  "state": "success", 
  "context": "netlify-cms/preview/deploy", 
  "description": "Deploy preview ready", 
  "target_url": "https://'"${BRANCH_NAME}"'.example.com/"
}

The state is required. The context is what I made up, but it is important to have the keyword "deploy" in there, as that's what Netlify CMS is looking for. The description can be whatever. The target_url is the preview link we are after!

Netlify CMS config.yml

If you want the preview link to point to the draft article itself, you will need to add a preview_path in the collection in config.yml. For example, preview_path: blog/{{slug}}.

Wow, all these, just for a link. Oh well, without this link, your writers and editors will probably be mad at you, so it definitely worth it.

How to enable SAML SSO with self-hosted gotrue and git-gateway

2020-04-03T06:18:46+00:00

Static Site Generator (SSG) is in vogue for building Web sites. Compared with traditional content management system (CMS), SSG is more performant, flexible and easier to maintain for people comfortable with coding . However, for the non-technical crowd, a WYSIWYG online editor is still the way to go. Netlify CMS fills this need nicely by offering a user interface for SSG. This article details how to enable single sign-on (SSO) for Netlify CMS, so it can be used in a business environment, where marketers, copy writers or other non-coding editors can edit and publish content.

Netlify's cloud platform offers SSO, and they also makes their software available in open source. Obviously, these open source software are not particularly well documented, so most people should be using their cloud offering. However, for some companies that are paranoid about privacy and business control, it is possible to host their own, for example, in order to utilize an existing corporate security infrastructure.

In this guide, we are going to utilize three Netlify's open source components to achieve SSO for a SSG powered Web site: gotrue is an identity service provider; git-gateway enables people to commit content in git without knowing about git; and Netlify identity widget provides the Web form for login.

We will connect gotrue with an existing Identity Provider (IdP) over SAML 2.0 protocols. In this case, an installation of Keycloak will be our IdP.

Keycloak Setup

First, we need to create a new client for the service provider (SP) in Keycloak. In this case, SP is gotrue, which is an identity API server written in go. In the administrative console of Keycloak, create a client with the ID https://example.com/.netlify/identity/saml, which points to the API endpoint of gotrue. Replace example.com with your domain name.

In the settings, make sure to uncheck Force POST Binding. Otherwise, gotrue will throw an error:

{"code":400,"msg":"Unsupported provider: No valid SSO service found in IDP metadata"}

Turns out that gotrue only supports HTTP-Redirect binding and does not support POST binding as of this writing. I find gotrue's design choice strange, as POST has much less uncertainty to deal with in a complex environment where a bunch of Web proxying and redirecting are going on. Most SAML SPs I dealt with use POST. Anyway, it is what it is.

Correspondingly, set Assertion Consumer Service Redirect Binding URL instead of the POST one. The value should be https://example.com/.netlify/identity/saml/acs.

On the other hand, Keycloak also has its own quirk. In its metadata API endpoint, namely {root}/auth/realms/{realm}/protocol/saml/descriptor, it returns an XML document that is not conforming to the SAML specification. A non-forgiving SP implementation will have a problem consuming the document, which is the case with gotrue. It will dump this error message:

{"code":400,"msg":"Unsupported provider: Fetching metadata failed: expected element type \u003cEntityDescriptor\u003e but have \u003cEntitiesDescriptor\u003e"}

As you can see, Keycloak's XML has an unnecessary extra layer of tag EntitiesDescriptor around EntityDescriptor, which trips gotrue. Strangely, RedHat decided not to fix this. See this issue for a fun conversation.

So you will have to fix it yourself. What I did was to manually copy out the correct metadata, save it in a XML file and put it on some public facing Web server temporally for gotrue to fetch. The correct metadata can be obtained in the Keycloak UI, go to "Installation" tab, then select "SAML Metadata IDPSSODescriptor".

Gotrue and git-gateway Setup

There is an excellent guide on how to self-host gotrue and git-gateway at https://github.com/hfte/netlify-cms-with-selfhosted-gotrue-and-git-gateway

Follow that guide first, and make sure it works. It does email based login, useful for inviting external bloggers or editors to work on the site. For SAML based SSO that is suitable for employees, read on.

There is a bug in the current version of gotrue source code. You will have to patch it first. On line 130 in api/provider/saml.go, we will need to add a NameIdFormat field. Otherwise, gotrue will send "" as the NameIdFormat, which cause Keycloak to error out with

"We are sorry... Unsupported NameIdFormat".

The fix should look like this:

        sp := &saml2.SAMLServiceProvider{
                IdentityProviderSSOURL:      ssoService.Location,
                IdentityProviderIssuer:      meta.EntityID,
                AssertionConsumerServiceURL: baseURI.String() + "/saml/acs",
                ServiceProviderIssuer:       baseURI.String() + "/saml",
                SignAuthnRequests:           true,
                AudienceURI:                 baseURI.String() + "/saml",
                IDPCertificateStore:         &certStore,
                SPKeyStore:                  keyStore,
                NameIdFormat:                "urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress",
                AllowMissingAttributes:      true,
        }

Here we set name id format to be email address, which apparently is the only format that gotrue works with. If you set the format as 'unspecified', users will be able to login, but they cannot edit anything, all the Netlify CMS UI controls will be greyed out. Obviously gotrue is checking that it gets an email address from the IdP.

After the patch, compile the code.

make deps
make build

Now the main configuration of gotrue is in a .env file in the root of the gotrue source code.

GOTRUE_JWT_SECRET="your-secret-key-shared-between-git-gateway-and-gotrue"
GOTRUE_JWT_EXP=3600
GOTRUE_JWT_AUD=localhost
GOTRUE_DB_DRIVER=mysql
GOTRUE_JWT_DEFAULT_GROUP_NAME=admin
DATABASE_URL="gotrue:mysqlpassword@tcp(mysqlipaddress:3306)/gotrue?parseTime=true&multiStatements=true"
GOTRUE_API_HOST=localhost
PORT=8081
GOTRUE_SITE_URL="https://example.com/"
GOTRUE_LOG_LEVEL=DEBUG
GOTRUE_OPERATOR_TOKEN=your-super-secret-operator-token
GOTRUE_DISABLE_SIGNUP=false
GOTRUE_MAILER_AUTOCONFIRM=true
GOTRUE_EXTERNAL_SAML_ENABLED=true
GOTRUE_EXTERNAL_SAML_METADATA_URL="https://idp.example.com/auth/realms/example.com/protocol/saml/descriptor"
GOTRUE_EXTERNAL_SAML_API_BASE="https://example.com/.netlify/identity/"
GOTRUE_EXTERNAL_SAML_NAME="Example SSO"
GOTRUE_EXTERNAL_SAML_SIGNING_CERT=''
GOTRUE_EXTERNAL_SAML_SIGNING_KEY=''
GOTRUE_EXTERNAL_LABELS="{SAML: Example SSO}"

The GOTRUE_EXTERNAL_SAML_METADATA_URL value should point to the corrected XML mentioned above.

Note, I set the default JWT group name to be "admin", because my IdP is backed by a LDAP server that handles user group membership and permisions already, there's no need to duplicate things here. Your case may be different.

Now the rest of gotrue and git-gateway setup is the same as the guide linked above.

Setup Netlify CMS

To add Netlify CMS to your site, follow the instruction

To get a nice login form with a SSO button for Netlify CMS, use netlify-identity-widget. This is done by adding some code in /admin/index.html of your SSG site. Mine looks like this, and it is a bit simpler than the guide linked above.

<!doctype html>
<html>
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Content Manager</title>
  <script type="text/javascript" src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script>
</head>
<body>
  <script>
    netlifyIdentity.init({
        APIUrl: "https://example.com/.netlify/identity"
    });
  </script>
  <script src="https://unpkg.com/netlify-cms@^2.0.0/dist/netlify-cms.js"></script>
  <div data-netlify-identity-button>Log in</div>
</body>
</html>

You may also want to add this line <script type="text/javascript" src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script> into the index.html of your main site. As the email confirmation link is pointed there.

Again, the rest of the setup should follow https://github.com/hfte/netlify-cms-with-selfhosted-gotrue-and-git-gateway.

Now your company personnel can use their company SSO credential to login to your SSG site to edit and publish content. No need to create another set of user name and password just for the Web editing work.

Migrate this blog from Drupal to a static site generator

2020-03-10T19:34:39+00:00

This blog has undergone a few migrations over its 15 years life span.

It started out as a Blogger account in 2005. The purpose was to record some system administration details when I was an assistant professor at university, setting up my laboratry for students.

Then Yunyao joined in after I moved the blog to a self hosted Drupal 5 installation in 2007. It has been with the same hosting company ever since, and I upgraded Drupal all the way to version 8.

But this old hosting account started to show limitations a couple of years ago. The technical specification simply could no longer keep up with the demands of modern PHP applications, which universally require the use of composer as a dependency manager.

Composer always ran out of memory in this very modest environment (512M memory limit). It took some effort to keep Drupal up to date without the modern tools. The final straw is an error that prevented us from creating new posts. It's likely a database error, but I could not investigate properly without tools such as drush.

So it is finally the time to move on. A static site generator (SSG) seems to be a good fit. Since the host only needs to serve static files, the limitation with my host account is not a concern any more.

Jekyll, Hugo and Pelican

There are perhaps hundreds of options for SSG. After some reading, I tried Jekyll, Hugo and Pelican. In the end, Pelican is the one that I got it to work, so here we are.

The first problem I needed to solve is to get the content out of Drupal and feed them into the SSG system.

Both Jekyll and Hugo have importers for Drupal 6 and 7 that directly import from the Drupal database. However none of them work with Drupal 8, which has some significant database schema changes.

I had some trouble compiling Ruby mysql library needed for Jekyll importer, so I gave up early on Jekyll.

On to Hugo, I quickly realized that I did not have the time to figure out the intricacy of the Drupal 8 database schema to adapt its Drupal 7 importer to 8. By the way, I also dislike golang the language, for I think it is reactionary (but that's for another post), so I gave it up.

On the other hand, Pelican has an RSS feed importer, which I easily modified the python code to extract the pieces needed for the frontmatter of Pelican. Since each Drupal installation is different, the resulting HTML could be very different. Having an easy to hack importer helps a lot.

The lesson seems to be that a feed importer is better than a database importer for a blog platform, since all the useful information are in the feed. There should not be a need to bother with databases, which tend to evolve a lot from one version to another. Feed format is universal.

Pelican and NetlifyCMS

After some minor cleanup of the markdown files generated from the feed importer, I quickly switched the live site to Pelican. I then pushed the whole thing to a github repository and thought I was done. Then Yunyao said she wanted a Web based editor like before, instead of having to write blogs like writing programming code.

That means we need a headless CMS to pair with Pelican. Again, there are so many options, and I just picked the first one that come into my head, NetlifyCMS. It includes simply two files that you drop into the content directory under /admin. It provides a react.js app with a nice editor for writing the posts in WYSIWYG fashion.

The form widgets for the frontmatter fields are a big help. Otherwise, I will have to copy an old post as the template to write new ones. The configuration of these widgets in a config.yml file is quite nice and easy.

backend:
  name: git-gateway

local_backend: true

media_folder: "content/images/uploads" # Folder where user uploaded files should go
public_folder: "/images/uploads"

collections: # A list of collections the CMS should be able to edit
  - name: "blog" # Used in routes, ie.: /admin/collections/:slug/edit
    identifier_field: "Title"
    label: "Article" # Used in the UI, ie.: "New Post"
    folder: "content" # The path to the folder where the documents are stored
    extension: md
    format: frontmatter
    create: true # Allow users to create new documents in this collection
    slug: "{{slug}}" # Filename template, e.g., title.md
    fields: # The fields each document in this collection have
      - {label: "Title", name: "Title", widget: "string"}
      - {label: "Date", name: "Date", widget: "datetime"}
      - {label: "Author", name: "Author", widget: "select", default: "Yunyao", options: ["Yunyao", "Huahai"]}
      - {label: "Category", name: "Category", widget: "select", default: "experience", options: ["experience", "notebook", "opinion"], hint: "Select category"}
      - {label: "Tags", name: "Tags", widget: "string", hint: "Enter comma-separated words"}
      - {label: "Series", name: "Series", widget: "string", required: false, hint: "Give multiple related articles a series name"}
      - {label: "Language", name: "Lang", widget: "select", default: "en", options: ["en", "zh"], hint: "Select en for English, zh for 中文"}
      - {label: "Status", name: "Status", widget: "select", default: "published", options: ["draft", "published"], hint: "Draft does not show on site"}
      - {label: "Body", name: "body", widget: "markdown"}

I am also taking advantage of its beta feature: local development. So I can write a bash script to write-blog:

#!/bin/bash

cd -- "$(dirname "$0")"

set -m 

git pull 

npx netlify-cms-proxy-server &
echo $! > npx.pid

invoke livereload &
echo $! > invoke.pid

echo "------------------------------------------------------------"

echo "Go to https://localhost:8000/admin to edit blog"
echo "Go to https://localhost:8000 to view blog"

echo "Don't forget to run 'publish-blog' to publish the blog to yyhh.org"

echo "------------------------------------------------------------"

fg

and to publish-blog

#!/bin/bash

cd -- "$(dirname "$0")"

kill -9 `cat npx.pid`
kill -9 `cat invoke.pid`

git add -A
git commit -am "publish blog"
git push

invoke publish

echo "Blog published at https://yyhh.org"

Good and Bad

This setup works nicely. Obviously, the site loads much faster, as it just serves static files.

Pelican also gets me some new features that I did not have before. The i18n subsite feature allows us to separate English and Chinese content completely, so people do not have to read things they cannot understand. The translation configuration with gettext and po files works pretty well.

Pelican is pretty old in term of "the latest and the greatest" technology fad, so there is not a lot of activities going on. There are not many themes that are actively updated. It is however sufficient for a personal blog.

The main concern is the slowness of the build. It would take about 50 seconds to build this site on my laptop (albeit a very old one). As more content is added, it would get worse. So I will definitely switch again in the future.

On the other hand, with the help of NetlifyCMS, the authoring is not affected too much by the slow build, because the CMS already gives us immediate feedback on the page we are editing. The build is only kicked off when we hit the publish button. So for now, I am OK with this setup.

Books Read in 2020

2020-01-02T05:29:00-08:00

I challenge myself to read or listen to 52 books in 2020. I'm listing them here with a short descriptions about them to keep track of what I've read and key ideas learned.

1. The Minimalist Home: A Room-by-Room Guide to a Decluttered, Refocused Life As a minimalist at heart, this is a great book for me to read (or rather to listen to). It provides inspiring stories as well as practical tips. By minimizing our life (it means different things to different people), we can focus on the things that are actually important to us.

2. Drive: The Surprising Truth About What Motivates Us This book talks about Motivation 3.0 and discusses the three critical elements of Motivation 3.0: Autonomy, Mastery, and Purpose. As a computer scientist in a world-class research lab, I completely agree with the importance of intrinsic motivation for highly creative work for myself and those I work with. It is really important for organizations and individuals to understand how to provide such an environment at work and at home. The author also provides good reference materials online worthwhile to check out.

The Gifts of Imperfection This took talks about how to lead a wholehearted and lives a meaningful life based on who you are, not you are supposed to be.

Migrate DokuWiki to another server

2018-10-20T04:46:00-07:00

DokuWiki is one of the most easy-to-use open source Wiki software. It is a very good internal documentation tool for small or medium sized organizations.

Comparing with using Google Docs for the same purpose, one advantage of using a Wiki is that it is more searchable and navigable. In addition, Wiki software is often very extensible. In the case of DokuWiki, there are hundreds of plugins that can help with many aspects of doing documentation work. In my company, we not only use DokuWiki to keep technical documentations up to date , but also used it for administrative chores, such as filling out forms, keeping track of vacations, and so on.

On the technical side, DokuWiki is a PHP application that does not reply on a database backend. All the materials are in plain text files. So one of the claimed advantages is that it is easy to migrate a DokuWiki installation from one host to another, because you could simply zip up the file directory, move to another host and unzip. As it happened, as part of my company's migration from AWS to Google Cloud, I had a chance to test this claim for real.

It turned out it was not as straightforward as DokuWiki claims. Although in the end, it was really a very simple migration, after I figured out the proper steps.

At first, I did what the document suggested: simply moved the files over to another host, but the site failed to load. Pouring over the error messages in the logs, I realized that some of the installed plugins could not compile, but somehow they did not break the old site. After removing these broken plugins on the old site, I decided not to simply zipping up the file directory again. Instead, I did the following:

1. Use a plugin called "Backup Tool" to zip up only essential configs and settings.

2. Download the latest stable version of DokuWiki software bundle, unzip on the new host.

3. Unzip the backup bundle created in step 1 over the fresh directory of DokuWiki created in step 2

4. Load the site, success!

5. However, the site reports some missing directories, such as media_attic and media_meta. Simply copying them from the old host to the new host fixes the problems.

6. Mission accomplished!

I think the main problem with the "copy files over" approach is that many caches on the old site tend to mess things up. A fresh new start is a surer way to migrate.

Backup Discourse with External PostgreSQL Server

2018-01-21T05:58:00-08:00

Discourse is a modern forum software that is quite popular in the technology circle. One can install a Discourse server easily with the recommended method of using docker. All the services needed by the Discourse server, e.g. Postgresql and Redis, will be running inside a docker container, which is fine for a small installation. However, if one has already an external Postgresql server running, e.g. on AWS RDS, and would like to use that instead, Discourse may have trouble doing backups, and you may receive an email from Discourse:

[2018-01-21 03:39:44] pg_dump: server version: 9.6.5; pg_dump version: 9.5.10

[2018-01-21 03:39:44] pg_dump: aborting because of server version mismatch

[2018-01-21 03:39:44] EXCEPTION: pg_dump failed

The main problem is that the Postgresql client in Discourse docker image is old, currently at version 9.5, whereas most of the world has moved on to version 9.6, and some even to version 10.

The Discourse people are not very helpful on their forum regarding this issue. So here's a solution:

We need to update the Postgresql version in the Discourse docker container to whatever version your external Postgresql server is. Fortunately, it is fairly simple. First, get into the running container:

sudo ./launcher enter app

Then update postgresql to the version you want, e.g.

apt-get install postgresql-9.6

Now link pg_dump to the right version:

ln -s /usr/lib/postgresql/9.6/bin/pg_dump /usr/bin/pg_dump

After this, you should be able to perform backup successfully in the UI.

LDAP Authentication for On-premise Sentry Server using freeIPA

2017-12-11T23:27:00-08:00

Sentry is a fairly popular service for tracking exceptions and errors in production softwares. They also provides a docker recipe for people who want to self host their own sentry server. This post shows how to enable LDAP authentication for such a self hosted sentry server, using freeIPA as the LDAP provider.

In addition to follow their instructions to install sentry, the following changes need to be made to add the capability to authenticate to sentry using freeIPA as the account source:

1. Change the Dockerfile to install some dependencies:

FROM sentry:8.22-onbuild

RUN apt-get update && apt-get install -y libsasl2-dev python-dev libldap2-dev libssl-dev
RUN pip install sentry-ldap-auth

2. Add the following code at the end of sentry.conf.py

#############
# LDAP auth #
#############

import ldap
from django_auth_ldap.config import LDAPSearch, GroupOfUniqueNamesType

AUTH_LDAP_SERVER_URI = 'ldap://ipa.example.com'
AUTH_LDAP_BIND_DN = 'uid=jenkins,cn=sysaccounts,cn=etc,dc=example,dc=com'
AUTH_LDAP_BIND_PASSWORD = 'secret'

AUTH_LDAP_USER_SEARCH = LDAPSearch(
    'cn=users,cn=accounts,dc=example,dc=com',
    ldap.SCOPE_SUBTREE,
    '(uid=%(user)s)',
)

AUTH_LDAP_GROUP_SEARCH = LDAPSearch(
    '',
    ldap.SCOPE_SUBTREE,
    '(objectClass=groupOfUniqueNames)'
)

AUTH_LDAP_GROUP_TYPE = GroupOfUniqueNamesType()
AUTH_LDAP_REQUIRE_GROUP = None
AUTH_LDAP_DENY_GROUP = None

AUTH_LDAP_USER_ATTR_MAP = {
    'name': 'cn',
    'email': 'mail'
}

AUTH_LDAP_FIND_GROUP_PERMS = False
AUTH_LDAP_CACHE_GROUPS = True
AUTH_LDAP_GROUP_CACHE_TIMEOUT = 3600

AUTH_LDAP_DEFAULT_SENTRY_ORGANIZATION = u'Sentry'
AUTH_LDAP_SENTRY_ORGANIZATION_ROLE_TYPE = 'member'
AUTH_LDAP_SENTRY_ORGANIZATION_GLOBAL_ACCESS = True
AUTH_LDAP_SENTRY_SUBSCRIBE_BY_DEFAULT = False

SENTRY_MANAGED_USER_FIELDS = ('email', 'first_name', 'last_name', 'password', )

AUTHENTICATION_BACKENDS = AUTHENTICATION_BACKENDS + (
    'sentry_ldap_auth.backend.SentryLdapBackend',
)

# optional, for debugging
import logging
logger = logging.getLogger('django_auth_ldap')
logger.addHandler(logging.StreamHandler())
logger.addHandler(logging.FileHandler('/tmp/ldap2.log'))
logger.setLevel('DEBUG')

LOGGING['overridable'] = ['sentry', 'django_auth_ldap']
LOGGING['loggers']['django_auth_ldap'] = {
    'handlers': ['console'],
    'level': 'DEBUG'
}

Some notes:

We are re-using a freeIPA system account that was created for read-only access (originally for Jenkins).
freeIPA use a flat structure for users: 'cn=users,cn=accounts,dc=example,dc=com'
AUTH_LDAP_DEFAULT_SENTRY_ORGANIZATION must be an exact (case sensitive) match with the organization full name, otherwise the logged in user will not have access to anything. The default organization name is "Sentry", but it can be changed in the UI.
Those logging statements are useful for testing and debugging

Updates [10/19/2020]:

This works up to sentry 9, for new versions of sentry, see this

Configure Jenkins to use FreeIPA LDAP Security Realm

2017-12-08T00:02:00-08:00

The point of setting up freeIPA for an intranet is to enable single-sign-on (SSO) for all the internal services that requires authentication and authorization. LDAP, originated from my alma mater University of Michigan, is one of the most widely accepted solutions to the problem. freeIPA can serve as a LDAP authentication and authorization provider to integrate with most of today's reputable server software. Jenkins is no exception.

There are plenty of guide of integrating OpenLDAP or other LDAP providers with Jenkins. However, there are not many guides on the particulars of integrating freeIPA with Jenkins. Here's how I got it to work.

Prepare freeIPA server

First, we need to create an LDAP account for Jenkins to access the LDAP data. Create a file, e.g jenkins.ldif

dn: uid=jenkins,cn=sysaccounts,cn=etc,dc=example,dc=com
changetype: add
objectclass: account
objectclass: simplesecurityobject
uid: jenkins
userPassword: secret
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0

Now use it:

$ ldapmodify -h ipa.example.com -p 389 -x -D "cn=Directory Manager" -W -f jenkins.ldif

Configure Jenkins

Go to Manage Jenkins -> Configure Global Security -> Security Realm, and choose LDAP, and set the following:

Server: ldap://ipa.example.com
root DN: dc=example,dc=com
User search base: cn=users,cn=accounts
User search filter: uid={0}
Group search base:
Group search filter:
Group membership: Search for LDAP groups containing user
    Group membership filter: (| (member={0}) (uniqueMember={0}) (memberUid={1}))
Manager DN: uid=jenkins,cn=sysaccounts,cn=etc,dc=example,dc=com
Manager Password: secret

Then click on Test LDAP settings and try login with an account, if results are all green, authentication is configured. Otherwise, try tweak the settings.

Now on to authorization, pick any one of the strategies. For testing, pick Anyone can do anything, so we will not be locked out. Once tested, I chose Matrix-based security, which give fine controls.

Once saved, one has to login to Jenkins with the SSO account of freeIPA, but that's the point, isn't it.

Caution

Once freeIPA is setup, it takes over the SSH sever and known_hosts file will not be updated in the account's .ssh directory. Instead, /var/lib/sss/pubconf/known_hosts is updated when ssh into another machine. This creates a bit of problem for setting up SSH based Jenkins slave when using Known host file Verification Strategy. A simple solution is to just copy the relevant host entry over.

FreeIPA in AWS EC2

2017-12-07T23:34:00-08:00

FreeIPA is the open source version of RedHat's identity management solution, which nicely integrates several open sources services that are important for managing an intranet: 389 LDAP Directory Server, MIT Kerboros, NTP, DNS, SSSD and others.

Most of my servers are virtual machines in AWS EC2. To manage such a cloud based intranet using freeIPA, some additional configuration is necessary. Here's how I got it to work.

DNS

The main problem of enabling freeIPA in EC2, is that every machine in EC2 has at least two kinds of of IP addresses. One is internal to the VPC only, e.g. the default VPC use IP addresses starting from 172.31.*.*; Another kinds of IP addresses are public IP addresses, which are different from the internal ones. A default install of freeIPA server and clients in EC2 will not work due to this dual IP addresses.

To install freeIPA, we first need to configure individual hosts' /etc/hosts, /etc/hostname files, so they point to the full qualified DNS name of the hosts. After that, we are ready to add these names to DNS servers.

Route53

We will bypass freeIPA's own DNS services, and use AWS Route53 DNS service. We need to setup three hosted zones for our network. One zone for the external IPs, one for internal IPs, and finally one for reverse lookup.

For internal and external hosted zones, in addition to the A records that map DNS names to IPs, we also need to add TXT and SRV records that allow freeIPA to discover services. Eg. for the external zone:

_kerberos.example.com.

TXT

"EXAMPLE.COM"

300

_kerberos-master._tcp.example.com.

SRV

0 100 88 ipa.example.com.

300

_kerberos._tcp.example.com.

SRV

0 100 88 ipa.example.com.

300

_kpasswd._tcp.example.com.

SRV

0 100 464 ipa.example.com.

300

_ldap._tcp.example.com.

SRV

0 100 389 ipa.example.com.

300

_kerberos-master._udp.example.com.

SRV

0 100 88 ipa.example.com.

300

_kerberos._udp.example.com.

SRV

0 100 88 ipa.example.com.

300

_kpasswd._udp.example.com.

SRV

0 100 464 ipa.example.com.

300

_ntp._udp.example.com.

SRV

0 100 123 ipa.example.com.

300

ipa.example.com.

99.99.99.99

300

Here we will install the freeIPA server on a machine with external IP 99.99.99.99, and the DNS name for the server is ipa.example.com.

Similar records need to be added the internal zone as well, just use the internal IP addresses.

Finally, the private reverse look up zone, named 31.172.in-addr.arpa., has records like these:

88.123.31.172.in-addr.arpa.

PTR

ipa.example.com

300

Where 172.31.123.88 is the internal IP address of the freeIPA server.

We need to do these for all servers managed by freeIPA. It's a bit of work if there are not many machines. For large deployment, one may want to investigate automatized solution with AWS APIs.

Test DNS

On a machine outside the VPC

$ dig +short ipa.example.com

Should return the external IP of the machine.

Doing the same on an internal machine should return the internal IP of the machine.

Finally, test reverse lookup on an internal machine

$ dig +short -x  172.31.123.88

Should return the DNS name of the machine.

FreeIPA Server Install

I normally use Debian servers, but there's currently no stable freeIPA server available in Debian Stretch, so I installed a Fedora, which supports freeIPA natively.

Use a small EC2 instance that will be dedicated to running a freeIPA server.

# yum install freeipa-server
# ipa-server-install

And say "no" to DNS. The installation should be successful if all instructions are followed.

FreeIPA Client

Since most of my machines are Debian, I had to install Debian freeIPA clients on them. Ubuntu Xenial universe repo has a version of freeIPA client that is compatible with Debian Strech. So I installed them.

# apt install freeipa-client
# /etc/init.d/ntp stop
# ipa-client-install

Notice that we must stop NTP daemon first if it's already running. Otherwise, the client installation will fail, because the freeIPA client expects to run its own NTP service that synchronizes with the freeIPA server.

After a successful installation, the client is still not ready to use, because the Ubuntu installer configured /etc/sssd/sssd.conf is currently broken: nss, pam, and ssh needs to be added. Otherwise, the client cannot be connected to. A working example of /etc/sssd/sssd.conf looks like this:

[domain/example.com]

cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = example.com
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = aclient.example.com
chpass_provider = ipa
ipa_server = _srv_, ipa.example.com
ldap_tls_cacert = /etc/ipa/ca.crt
[sssd]
services = nss, sudo, pam, ssh

domains = example.com
[nss]
homedir_substring = /home

[pam]

[sudo]

[autofs]

[ssh]

[pac]

[ifp]

[secrets]

Restart sssd or simply reboot. Everything should work as expected.

$ kinit admin
$ ssh admin@ipa.example.com

Should login to the freeIPA server as admin user.

$ ssh admin@aclient.example.com

Should login to the client machine as admin user.

Congratulation, now you have single sign on (SSO) for your intranet in AWS EC2!

[update: 10/23/208]

Cross Cloud Intranet

With minor modification, this same setup can be used to manage you Intranet with hosts spanning multiple cloud platforms!

For example, you can have some hosts reside on Google Cloud Platform (GCP) while the IPA server lives in AWS. To do that, the public IP addresses of these ex-AWS hosts need to be entered in both external and internal DNS realms in Route53. GCP nicely supports this setup because you can reserve as many static public IP addresses as you want in GCP.

For example, you can have a host on GCP with a public address 35.22.31.33, for which you assign a domain name "gcp1.example.com" in Route53. Then you run hostname gcp1.example.com on this GCP host. After that, you should be able to install freeIPA client on it to enroll into your Intranet.

After install freeIPA client, another important modification, is to add the following directive in /etc/krb5.conf of all your ex-AWS hosts:

[libdefaults]
  ignore_acceptor_hostname = true

This directive tells Kerberos service of an accepting host to not verify its own hostname, because an ex-AWS host's attempt to discover its own hostname will yield a name that is different from the one you assigned in Route53. With this change, you should be able to access your ex-AWS hosts as if they are part of your Intranet. Your SSO should work on these ex-AWS hosts as well.

Ecobee3 installation with K wire

2017-12-01T06:13:00-08:00

As part of our house modernization process, we bought an Ecobee3 Wifi enabled smart thermostat. We hope that this small device will help reduce the size of our utility bill. The product looks very simple, but it comes with three additional sensors that can be placed in different rooms, and these can even detect room occupancy!

One problem I encountered was how to install this device when the old device used a K wire, which was not documented any where on Ecobee's site. See the circled K below:

My Internet search eventually brought me to a Youtube video, where someone had a similar problem with installing a Nest thermostat. Basically, the solution was to connect the K wire to the Y1 wire.

I said to myself, maybe this works for Ecobee3 too. So I did just that:

When powered up, Ecobee detected that Y1 wire is connected, and indicated that G must also be connected.

This could easily be done on the menu.

Now everything works. What a nifty little device!

Upgrade Drupal from 6 to 8

2017-11-21T23:46:00-08:00

Since Drupal 6 is no longer supported, I upgraded this site to the lasted version of Drupal 8.4.2 by following the guide. As you can see, the upgrade mostly worked. However, there are a few points of caution as well as some unresolved problems.

As before, I setup a test site in a sub-directory (h/drupal) of the main site (/h) and assigned a domain name to the test site. The test site use a new empty database. The idea is to keep the old site running, and migrate the data from the old to the new.

This blog is on a hosted service that allows direct SSH access, so it made things a lot easier. Drush helped a lot. To get drush to work, I had to modify the drush bootstrap script to use php7.1-cli, because the hosting server had many versions of PHP installed, and the default one is not even a command line interpreter. I then created a bash alias in ~/.bash_profile for the drush script, so that I could run drush anywhere.

alias drush='/h/drupal/vendor/bin/drush --root=/h/drupal'

With drush, migration was easy, with only a few commands.

First initialized the database.

drush si standard --db-url=mysql://username:pasword@new.mysql.server/newdb --root=/h/drupal --db-prefix=drupal_ --locale=en

Now we found out all the modules enabled on the old site, and enabled them on the new site.

drush en migrate_upgrade migrate_tools migrate_plus rules config_update libraries tracker views_bulk_operations taxonomy_menu tagadelic mollom better_formats statistics pathauto mathjax profile

Not all the old modules exist in Drupal 8 any more. Some of them are folded into core, and some simply disappeared. With above, we also enabled three modules needed for doing the migration.

First created the migrate configurations.

drush migrate-upgrade --legacy-db-url=mysql://username:password@old.mysql.server/olddb --legacy-root=https://yyhh.org --configure-only

At this point, one could run individual migration one by one, or one could run them all, which I did:

drush mi --all

Most of the migrations worked. Two migrations failed with errors:

upgrade_d6_filter_format

"Missing filter plugin: filter_null. [error]"

This error turned out not to be a problem. All one needed to do is to save the formats again at the UI: "/admin/config/content/formats". Otherwise, the content of the posts will not show due to missing filter "filter_null". Saving the formats in the UI got ride of the errors.

upgrade_d6_taxonomy_term_translation

"Drupal\Core\Database\IntegrityConstraintViolationException: SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'langcode' cannot be null:..."

Basically, all the taxonomy terms failed to translate due to missing language code. Since all my terms are in English, this was not a problem either.

Now the new site was up and running. The look of the site was of course horrible. I had to install a new theme and created a sub-theme. Then did all the work of creating views, blocks, and links with the UI. Now we have a functioning site. Cool!

A few cautions though:

Do not enable "taxonomy_breadcrumb", otherwise the page will error out. I think it's because the migrated taxonomy terms miss some fields. I did not investigate further since I would not use it any more.
Do not enable "comment_notify", the migration may error out. I did not investigate since there's no need to migrate this.
No need to enable "blog". The functionality of blog module could be easily reproduced by creating one's own views. The "blog" module does not work well with migrated posts any way.

An unsolved problem is that the "popular content" links are all out of whack due to the loss of all old statistics. This is a known issue that has not been resolved as of today. The fix, however, will be available in Drupal 8.5. So if the node counts are important to you, hold the migration until 8.5 is released on March 7, 2018.

Overall, the upgrade is a smooth experience, in the sense that source code modification was not necessary, nor was changing the database data. I am glad the Drupal is getting better and better.

Tips for Traveling with a Baby in Japan - Diaper Bag for Air Travel

2017-01-11T01:09:00-08:00

The list of items contained in our diaper bag for air travel, annotated with how essential each item was based on our experience as follows:

*** Must have

** Nice to have

* Not needed

For travel gear that may not fit in your diaper bag, please visit this page.

Portable changing pad (***)

This is definitely essential for both in the airports as well as on the airplane. We like our JJ Cole Collections Changing Clutch.

Diaper (***)

Double the amount you will usually need for the duration of the travel just to be on the safe side (e.g. in case of flight delay). We stuck in the Los Angels airport for an extra 4 hours on our way back and used up all the diapers that we thought we would not need.

Baby wipe (**)

Tips for Traveling with a Baby in Japan - Travel Gears

2017-01-09T06:50:00-08:00

We traveled with Boyan, our 13-month son last month to Japan (for those who are wondering, yep, he is the main reason why this blog has been so quiet in the past couple of years. :-)) and spent a unforgettable 10 days there. Before I went there, I searched over the web for tips, but they seemed to scattered around. I therefore decided to share our experience and hope it will be useful for others.

To make the information most useful, I will just list items we brought with us for the trip and comment on how essential each item was based on our experience denoted in the following way:

*** Must have

** Nice to have

* Not needed

**1. Travel Gears

Mountain Buggy Nano (***)

It's definitely a life saver to travel with a small child. The major cities in Japan are very stroller friendly, even though it seems that locals prefer baby carriers over strollers. We did bring a baby carrier with us but ended up using the stroller most of the time.

Pro:
- It is lightweight and can be folded into carry-on size.
- It's back can be adjusted into different positions, including a lie flat position for sleeping
- It has up to 44lb weight and children up to 4 years
Con:
- The sun shade is on the small side. But this issue has already been fixed for newer model of Nano.

Mountain Buggy Nano All Weather Cover (***)

Unless you plan to stay indoor all the time, the all weather covers are must-have to ensure your child stay comfortable all the time. We used the rain cover for both rainy and windy conditions. We also used the sun shade to create a dark environment to allow Boyan to take a good nap during the day.

Pro:
- Both covers fit into a small bag that can be attached to the stroller.

Hip Seat Baby Carrier (*)

We have multiple baby carriers, including Egobaby. But our favorite is Bebamour baby carrier, mostly used as hipseat. During our Japan trip, we used it from time to time, but only for short trips. However, we did notice that baby carriers are very popular for locals.

Hearing Protection (**)

I love my Bose Noisy Canceling headset for air travels. Even though Boyan was only 13-month, we got the large size Baby Banz earBanZ Hearing Protection and it worked great for him. He seemed to very comfortable in it and wared it all through the flights. He looks super cute in it as well. :-)

Getting a window seat on the airplane (***)

Not really a gear, but it is super important. It helps to keep the kid entertained from time to time, especially at boarding and landing time. More importantly, it allows the kid to sleep with the least amount of interruption.

Neck wallet and passport holder (***)

A must-have for traveling with a small child. It makes it so much easier to manage all the important documents and tickets on the go. The last thing you want is to leave any of the important documents behind.

Diaper bag (***)

There is no need to explain why it is critical to have a well-stocked diaper bag for travel, particular on the road in a foreign country.

I will write a separate post soon on the content of the diaper bag for both air travel as well as for day trips. Stay tuned.

Data-Oriented Programming (DOP)

2016-12-03T23:33:00-08:00

JSON is arguably the world's most popular human readable data format today. It has largely replaced XML as the data exchange format on the Internet. One of the key reasons for the proliferation of JSON is its simplicity. The data structure are very limited: only arrays, enclosed with []; and objects, enclosed with {}. That's it. It cannot be simpler.

Apparently, this dead simple data format is enough to represent the vast landscape of data that JSON becomes the de-facto data format for Web services. Most Web APIs we use today speaks JSON. However, JSON is not native for most programming languages. It becomes a pain to convert to and back from JSON in programming languages.

What if we develop a Data-Oriented Programming (DOP) language, that can speak something similar to JSON natively?

Luckily, this language already exists. It is called Clojure!

In Clojure, [] means the same thing as in JSON, {} means essentially the same thing as well: a key value map. The only thing added, which makes it a programming language instead of a purely data format, is a pair of (). What () enables is the abilities to define and call functions. With this additon, we get a fully general purpose programming language.

As you may have suspected, using () to define and call functions is what Lisp do. So you are right, in this sense, Clojure is a Lisp. But with the ability to handle data like in JSON, it is a Data-Oriented Programming language.

So, there you have it, Clojure is the first DOP language, a DOP Lisp.

Switching to Spacemacs from Vim for Clojure/ClojureScript Development

2015-08-13T06:00:00-07:00

Clojure has been my primary programming language for a couple of years now. During this period, I have relied on my trusty Vim text editor as the development environment. Coding Clojure in Vim had been an enjoyable experience with these excellent Vim plugins:

So, why am I switching? Well, Clojure is, after all, a LISP. The tooling coverage for Clojure in the LISP's natural habitat, Emacs, is simply more complete than in Vim. I did not realize the extent of the discrepancy until last week, when some guys in the Clojure meetup demonstrated impressive refactoring features of their Emacs Clojure development environment.

To be fair, fireplace is doing an adequate job of supporting the most of dynamic features necessary for Clojure programming: code evaluation, documents lookup, tests and so on. I have been productive with it in the last couple of years. On the other hand, as my projects grow bigger and more complex, better support for debugging and refactoring seems to become desirable. These capabilities exist in Emacs.

Of course, I am not about to give up the Vim style text editing. As a HCI researcher in my previous life, I know that theoretically, Vim style text editing is simply better than text editing with GUI and a mouse, because Fitts' Law is real and it hurts. In addition, the superiority of modal editor vi over non-modal editor emacs for text editing has been empirically established as far back as 19831.

For my case, an ideal situation would be to keep the text editing style of Vim, but use it in Emacs to get the benefit of extensive Clojure support. Not surprisingly, plenty of people have worked towards such solutions. The latest effort is in the form of spacemacs, a fantastic open source project that is enjoying an outpouring enthusiasm from the community, earning more than 3000 github stars in a very short time.

I had to try it. Try as I did. And I can say that I am not disappointed.

To be honest, the on-boarding process left a lot to be desired. First, on OSX, there's already a default installation of emacs, which would not work with spacemacs. The recommended homebrew installation of emacs is simple enough, but one needs to rename the default emacs and make the homebrew one the default. This is not mentioned in the guide.

I found configuring the editor to be surprisingly easy, considering I knew nothing about emacs before (other than the key combination to quit from one). The majority of the Vim key bindings I tried work as desired. The ones that did not work are not hard to change. Within a couple of hours, I have managed to create a configuration that replicates most of the key bindings of paredit.vim, which I need to be productive coding Clojure. Here are my dotspacemacs/config to achieve these and more.

(defun remove-background-color ()
  "Useful for transparent terminal."
  (unless (display-graphic-p (selected-frame))
    (set-face-background 'default "unspecified-bg" (selected-frame))))
(defun dotspacemacs/config ()
  "Configuration function.
This function is called at the very end of Spacemacs initialization after
layers configuration."
  ;; Make evil-mode up/down operate in screen lines instead of logical lines
  (define-key evil-motion-state-map "j" 'evil-next-visual-line)
  (define-key evil-motion-state-map "k" 'evil-previous-visual-line)
  ;; Also in visual mode
  (define-key evil-visual-state-map "j" 'evil-next-visual-line)
  (define-key evil-visual-state-map "k" 'evil-previous-visual-line)
  ;; clojure mode config
  (require 'clojure-mode-extra-font-locking)
  (add-hook 'clojure-mode-hook #'smartparens-strict-mode)
  (add-hook 'clojure-mode-hook #'evil-smartparens-mode)
  (add-hook 'clojure-mode-hook #'rainbow-delimiters-mode)
  ;; start a light theme when launched as GUI
  (when (display-graphic-p)
      (progn
        (disable-theme 'darkburn)
        (load-theme 'leuven t)
        (enable-theme 'leuven)))
  ;; remove background color for both server and client
  (add-hook 'window-setup-hook 'remove-background-color)
  (add-hook 'server-visit-hook 'remove-background-color)
  ;; remove trailing whitespace when saving
  (add-hook 'before-save-hook 'delete-trailing-whitespace)
  ;; toggle comments
  (define-key evil-normal-state-map ",c " " cl")
  ;; match paredit.vim key-binding
  (define-key evil-normal-state-map ",W" " kw")  ; wrap with ()
  (define-key evil-normal-state-map ",w["        ; wrap with []
    (lambda (&optional arg) (interactive "P") (sp-wrap-with-pair "[")))
  (define-key evil-normal-state-map ",w{"        ; wrap with {}
    (lambda (&optional arg) (interactive "P") (sp-wrap-with-pair "{")))
  (define-key evil-normal-state-map ",S" " kW")  ; splice, i.e unwrap an sexp
  (define-key evil-normal-state-map ",J" " kJ")  ; join two sexps
  (define-key evil-normal-state-map ",O" 'sp-split-sexp) ; split an sexp
  (define-key evil-normal-state-map ",I" " kr")  ; raise current symbol
  (define-key evil-normal-state-map (kbd ", <up>") " kE") ; splice kill backward
  (define-key evil-normal-state-map (kbd ", <down>") " ke") ; forward
  ;; These are different from vim, here cursor should NOT be on delimits
  (define-key evil-normal-state-map ",>" " ks")  ; forward slurp
  (define-key evil-normal-state-map ",<" " kS")  ; backward slurp
)

As can be seen, all these functionality of Vim have already been coded up by someone and included in spacemacs as functions, all I did was changing their key-bindings. That was easy :-). I am looking forward to the journey ahead with spacemacs.

1. Poller, M.F., Garter, S.K. A Comparative Study of Moded and Modeless Text Editing by Experienced Editor Users. In Proceedings of CHI '83, pp 166-170

Juji Personality Profile for Mayor Stubbs

2015-03-19T04:53:00-07:00

What makes Mayor Stubbs special?

Check out the personality (or shall I say, catality) profile for Mayor Stubbs constructed by Juji based on Mayor Stubbs' tweets.

Have you meet Mayor Stubbs? Does this sound like him?

What really makes me smile is the summary of the profile. Mayor Stubbs is "curious investigator". This is amazingly accurate: is there a cat that is not curious? I don't think so.

Does your pet have its own twitter or Facebook account? If so, check out its Juji profile or check out your own or your friends'. It's super fun.

Museum and Cultural Tour of Bulgaria in 9 Days - Day 2 (Boyana Church)

2014-04-05T05:05:00-07:00

After a long drive, the last stop of the second day of our trip is National Museum of Boyana Church, a UNESCO World Heritage Site. It is a very small church, originally built in the lat 10th or early 11th century, with the addition of the central wing in the 13th century.

It is famous because of its frescoes from 1259, which demonstrate the exceptional achievements of mediaeval Bulgarian culture. For obvious reason, photography is not permitted in the church. Here are some representative pictures from the Web (credits go to Wikipedia)

Museum and Cultural Tour of Bulgaria in 9 Days - Day 2 (Rila Monastery)

2013-12-13T22:39:00-08:00

The second day of our trip began with a long drive from Sofia to Rila Monastery, the largest and most famous Eastern Orthodox monastery in Bulgaria. It is the first UNESCO World Heritage Site in Bulgaria we visited in our trip.

This beautiful monastery is situated in the southwestern Rila Mountains. Founded in the 10th century, the Rila Monastery is regarded as one of Bulgaria's most important cultural, historical and architectural monuments. The monastery is named after its founder, the hermit Ivan of Rila (876 - 946 AD).

Beautiful mural

Bell Tower

Sneak peek of the Monastery from the window of the tower

Ancient mural inside the tower

Long line outside of the Monastery

A cat enjoying an afternoon nap near the Monastery before waked by cat-loving tourists coming to pet it.

A food stand nearby to grab a quick lunch.

Tip: Remember to bring your own water bottle. There is a fountain right outside of the Monestery to refill it with real spring water, helping save $$$ and our mother earth.

Museum and Cultural Tour of Bulgaria in 9 Days - Day 1 (More)

2013-09-21T18:22:00-07:00

We were quite amazed by the National Archaeological Museum, especially the Thracian treasures.

The museum is located in the largest and oldest former Ottoman Mosque in Sofia.

Thracian treasures. Can you believe that they were made before the Roman Empire?

Gold jewery

Gold wreath

Gold mask and ring from Thracian king's grave.

Museum and Cultural Tour of Bulgaria in 9 Days - Day 1

2013-09-08T04:31:00-07:00

(1) Sign of Sofia --- City of Lion.

How to get a kitty manicure

2013-09-04T05:34:00-07:00

Speaker: Tom

Dear fellow kitties, I am going to show you how to get a kitty manicure and pedicure.

First of all, let me show you my beautiful finger nails. My mom polishes each and every one of them on a weekly basis. Such weekly pampering is essential for a purrfect kitty like me.

Now, let me show you how to get a kitty manicure.

First of all, relax and get each of your finger massaged. I like to sit on my mommy's lap to do the massage, but Mayori prefers to get her massage in her favorite bed.

After the massage, you may feel a bit sleepy. This is a good time to get your manicure.

Continue relaxing and enjoy your pedicure.

It is just this simple --- just relax and enjoy! (^-^)

Museum and Cultural Tour of Bulgaria in 9 Days --- Preface (保加利亚文化历史九日游 - 前言）

2013-08-25T18:36:00-07:00

Planning:

Following my usual strategy for travel planning, I first got some rough ideas on the highlights of the country by referring to the "Best of Bulgaria in 12 Days Tour" by Steve Rick. Based on Huahai and my own schedule, I then decided that we could spend at most 9 days for the trip.

Usually, we would do self-driving tour for such a trip. However, I found out that it may not be very easy to go to some of the places on our own due to the language barrier. Given this potential issue plus our busy schedule due to work, I decided to use a tour company to help the planning. Fortunately, custom tours in Bulgaria are quite affordable. We used Traventuria (https://traventuria.com/), which have received great reviews on tripadvisor. Communications with the company was very easy. After a few email exchanges, we settled down on the following iternary. Traventuria helped us book all the hotels and also was responsible for providing transportation, including a private car and driver (who also served as our translator and tour guide).

Equipped ourselves a English guide book: Bulgaria (EYEWITNESS TRAVEL GUIDE), we were ready for our big trip!

BTW, I highly recommend this book, which turned out to be extremely helpful during the trip, as not every place we went to provides descriptions in English. In addition, we also found a couple of new places from the book to add to the original iternary (e.g. Stone Forest).

Iternary:

Day 0 Pick-up from Sofia airport and transfer to Hilton Hotel.

Day 1 Tour of Sofia (about 3 hrs) + Archaeological Museum.

Day 2 Rila Monastery + Buyala Church

Day 3 Etara open-air museum + Veliko Tarnovo + Tsarevets Fortress

Day 4 Rousse + Roussenski Lom

Day 5 Madara Horseman + Stone Forest + Varna

Day 6 Nessebar

Day 7 Plovdiv

Day 8 Bachkovo Monastery + Shiroka Luka + Devin

Day 9 Devils Throat Cave + Yagodina Cave (return to Sofia)

Day 10 Departure from Sofia airport

准备和安排：

这次选在保加利亚度假主要的原因是YY要在这里参加一个为期将近一周的学术会议, 还是安排一个旅游项目才对得起来回在飞机上消耗的差不多两天的时间和我们几个月前得的一年有效期的法国申根签证。

YY一向擅长懒人旅游攻略，绝不会花超过一两个周末的时间做功课。而这次YY懒人旅游攻略又上了一个新的层次。在参考了Steven Rick的十二天的保加利亚旅游路线之后，YY决定至少得有一周以上的假期，而且主要以看保加利亚的世界文化遗产为主。不过我们工作都挺忙的，所以给自己放一周假加上前后的周末也就差不多是极限了。YY又开始研究给HH定票的事情。八月是欧洲的旅游旺季，飞机票都其贵无比，而里程票根本就没有。幸好可以用积累了多年的信用卡的点数来换票还稍微便宜一些，但也花了快十三万点。

机票订下了以后，YY先在tripadvisor上找了一个最受好评的旅行社，让他们按以上要求安排一个合适的路线和提供估价。看了路线和估价后YY决定一切让旅行社包办，因为似乎不比自己安排贵多少，可是需要准备需要的事情和路上可能遇到的麻烦却少了很多。另外旅行社也提供自驾游，但是这里物价便宜而我们又语言不通上包辆车确实是省时省心的好办法，特别是我们要去的一些地方非常偏远，用GPS也不好找，在迷路的时候可能会让两人都很紧张。话说平时辛苦挣钱可不就是为了好好花吗？于是在几个电子邮件往来加上付款加上付款之后，行程就订下来了。YY从网上买了一本保加利亚旅游指南( Bulgaria (EYEWITNESS TRAVEL GUIDE)就算是做完了所有必须的准备工作了。

基本行程：

第0天：HH被从索非亚机场接到索非亚希尔顿酒店，和在这里已经开了一周会的YY会合

第1天：索非亚一日游

第2天：Rila修道院及Boyala教堂（住索非亚）

第3天：Etara民俗村, 查雷维茨皇宫(住大特尔诺沃)

第4天：鲁塞, Roussenski Lom

第5天：马达拉骑士(Madara Horseman)，石头森林（Stone Forest）瓦尔纳（住瓦尔纳）

第6天：内塞巴尔(Nessebar)一日游（住内塞巴尔）

第7天：普罗夫迪夫（Plovdiv）一日游（住普罗夫迪夫）

第8天： Bachkovo Monastery - Shiroka Luka ,Devin

第9天：Devils Throat Cave and/or Yagodina (回索非亚)

第10天：回家

My First Encounter with Dirty Diaper

2013-01-20T08:02:00-08:00

Today I have encountered dirty diaper for the very first time in my life and it is not pretty. While doing routine cleaning of our guest bathroom, I was shocked to discover a stack of dirty diaper deposited in the trash can in our guest bathroom. I am still struggling on how to get rid of the terrible smell of the trash can after dumping the diapers. We do not have anyone in need of diaper in the house.

The only plausible explanation is that some guests attended our house party a few days ago left them behind. I thought that all the books on how to raise kids include education on basic etiquettes when visiting other people's house. But obviously I was wrong!

Out of curiosity, I searched online to see what is the expected diaper etiquette and found one here. In summary,

No one leaves dirty diapers in a guest bathroom.

When visiting other people's house, one should "place the dirty diaper in a plastic bag (and twist-tie it shut), then place that bag inside your diaper bag, and ask your hosts where they would like you to dispose of it — presumably in a trash bin outdoors or an inside bin that’s emptied frequently. " Had my guests know and follow this protocol, I would not have to deal with the smelly trash can problem in the guest bathroom!

Free Airport Wi-Fi? -- Don't Count on It in Europe (Update on 08/25/2013)

2012-10-26T18:28:00-07:00

I was traveling in Europe last week. While I enjoyed the history, culture and food very much, I was not a big fan of the airports. Like many people, I would like to stay connected most of the time, at least be able to access my emails and the Web from time to time (say every 2-3 hours). Unfortunately, passing through the following three airports in Europe is almost like returning back to stone age, in terms of connectivities.

Initially, I thought that CDG was pretty bad --- it does have free Wifi, but only for 15 minutes. Then I have to buy Wifi Access. However, at least I was able to browse the airport website for free.

Then MAD really made me a bit mad ---- no any sort of Wifi whatsoever available, free or not!

Now, FRA does provide 30 minutes free Wi-Fi. That's better, right? However, then you need to be able to receive text on your cell phone to get the pin. Some genius obviously believes that everyone visiting FRA has a cell phone and every cell phone has service in FRA. Unfortunately, my cell phone does not work here; so no luck for me. :(

Of course, I can always spend a few Euro to buy Wi-Fi access. However, free Wi-Fi in airports is almost a norm in today's world, isn't it? Even in developing countries such as El Salvador has free Wi-Fi in its airports. Unfortunately, I don't really see the situation will likely to improve in the near future. This note is just a reminder to myself to load up my laptop and iPhone with things that I can work/read while in the airports off the grid.

Update on 08/25/2013:------------------------------------------------

It turns out that not all European airports are that bad. During my recent trips to Europe, I have discovered that SOF (Sofia airport in Bulgaria) provides unlimited free Wi-Fi. MUC (Munich airport) provides 30 minutes free Wi-Fi and requires only an email address. Even FRA seems to have updated their genious approach of providing free Wi-Fi as well and requires only an email address now. Things are looking up!

Fixing Problems after Upgrading from Snow Leopard to Mountain Lion

2012-09-24T00:28:00-07:00

The IT department of my company has been urging us Mac users to upgrade OSX to Lion a long time ago. After getting a few papers submitted last week, I finally got around to upgrade the Snow Leopard for my Macbook Pro work machine. Since I couldn't find Lion on Apple Store any more, I decided to go straight to Mountain Lion. The download and installation went smoothly, and most things seemed to work after the upgrade. Here are a few things that broke and the fixes I found.

SSH with public key

Mountain Lion changed a few things that broke password-free SSH access to and from OSX using public/private key pairs.

SSH from Mountain Lion to older SSH severs:

Mountain Lion upgraded openssh client to version 5.9p1. SSH to some older version of ssh server would not work ("Connection reset by peer") due to ciphers being too long. We can use a shorter one by adding to file "~/.ssh/config".

Host address\_of\_your\_ssh\_server  
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour

This problem is a known issue of openssh. I remember did this fix for my Linux machines.

SSH to Mountain Lion:

This one took me a lot of googling to find the cause. Basically, Mountain Lion changed /etc/sshd_config file, so that openssh server only checks "~/.ssh/authorized_keys" now, instead of checking both that and "~/.ssh/authorized_keys2". To fix this, all we need to do is to rename the later to the former.

$ mv ~/.ssh/authorized_keys2 ~/.ssh/authorized_keys

I find this change rather annoying, as it adds little benefit but creates a lot of troubles. For example, it broke my backup solution. I am using backuppc to backup this Macbook to a central backup server through password-less SSH. It took me a while to debug when backuppc reported the problem.

Homebrew

Running "brew doctor" would show the information needed to fix homebrew. Basically, one had to install xcode 4.5, install command line tools, install X, and so on. A lot of downloads and wait time.

Latex

All the latex programs were not in path any more. Since my latex distribution was pretty old anyway. I chose to download the latest version of MacTex to install. It took a long time to download though.

Lotus Notes

Mountain Lion has its own Notes.app now, which conflicts with Lotus Notes. During upgrading, Lotus Notes will be moved to be under a directory "/Application/Lotus Notes Local" or something similar. All you need to do is to find Notes.app in there, and rename it something else, e.g. "LNotes.app", and move it back to be under "/Application".

Java

Java is not installed by default. When using applications require it, you will be prompted to install.

In conclusion, the upgrade from Snow Leopard to Mountain Lion works reasonablly OK, but be prepared to fix some problems. I think the upgrade experience of OSX is not better than Linux, as both require a similar amount of tweakings and searching solutions. However, upgrading in OSX costs money. If this was not on company tab and by their urging, I would not have bothered.

Pursue of the Perfect Solution for Cat Hair (1) --- Vacuum

2012-08-27T00:40:00-07:00

If you have one or more long hair cats at home, like our Mayori and Tom, you probably share the same dream as us --- a house without cat hair, at least a house without having cat hair everywhere.

Towards this dream, we have tried out various products. Here is a list of the various vacuums we have used.

1. Hoover WindTunnel T-Series Rewind Upright Vacuum

It work pretty well on carpet, but it is not good for hard surface like hardwood floor --- the air blowing out of it makes the cat hair flying everywhere. It's bagless, so cleaning it is not a very pleasant task. Fortunately, Huahai is usually the one doing the cleaning work. 😛

Overall, this is a cheap and durable vacuum. Ours lasted for over 4 years with at least weekly usage. I would recommend getting it if you live in an all-carpet place or plan to use it in combination of another vacuum for hardwood/tile floor.

2. Eureka Hand-Held Vacuum

As indicated by many online reviews, this is a surprisingly powerful vacuum given it's size. I brought it to do some quick cleaning job. However, I don't really like it that much --- it's not cordless, so using it for quick cleaning is not particularly convenient. It is too heavy and too noisy for me to do any thorough cleaning job. We have already got rid of it after having it sitting in our garage for over a year.

3. Electrolux Ergorapido Bagless Cordless Handheld/Stick Vacuum Cleaner

I like this vacuum a lot --- it is lightweighted and dual-usage. This is the one we have been using for nearly 5 years for quick cleaning job. (BTW, if you are Costco member, you can get it for lower price from Costoc than from Amazon.com, especially when it is on sale with coupon).

4. Miele Olympus S2120 Canister Vacuum Cleaner

This one is expensive but works like a dream, especially for cat hair as the air is blowing from the top. It is very powerful yet very quite. Great for both hardwood floor and carpet (with the Miele Floor Brush). This will be the vacuum that we are going to use for many year to come.

5. iRobot Roomba 770

Even with Miele, we still need additional help --- Roomba is the perfect solution for busy people like us. If you had an old Roomba before and found cleaning it a pain, please take another look at it --- the cleaning tools coming with the newer generations of Roomba makes the cleaning much easier than before. This one is also a keeper.

High Resolution Image for Pdflatex

2012-08-09T20:31:00-07:00

While preparing the camera-ready version for our CIKM demo, my colleagues and I found that the screenshot included in our paper appeared to be fairly blurry in the pdf version, even though the original .jpg file looks fine.

I searched online to try to find the solutions. After a few trials and errors, adding the following two lines into the beginning of the latex files (before \begin{document}) did the trick.

\pdfpxdimen=1in

\divide\pdfpxdimen by 300

The above solution basically forces the pdf file generated at a higher resolution (in this case, 300 dpi) so that everything, including the images, looks better.

The original solution comes from here.

Screen Saver Option Disabled on Windows 7/XP

2012-08-01T17:07:00-07:00

For unknown reason, my machines would suddenly disable their screen saver options (e.g. with the "on resume, display log on screen" check box grayed out). Whenever this happens, it would trigger the security alert and I will get a warning from the IT department to fix the issue.

I am tired of having to hunting down the answer every time and therefore I am recording it here.

The following method is what usually works for me.

1. Go to "Start" type in "regedit"

2. Go to

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Windows\Control Panel\Desktop]

Delete the "ScreenSaverIsSecure" value.

Machine Learning That Matters

2012-07-26T21:20:00-07:00

I happen to come across this excellent article from this year's ICML on Machine Learning that Matters. I encourage everyone, especially computer science Ph.D students, to read it.

As an industrial researcher who draws a lot of job satisfaction by making both scientific and practical impact, I completely agree with the author on that we need to work on research "that matters", not just for machine learning. Many of the challenges outlined by the author are applicable to many other areas in computer science as well.

If you have ever encounter questions from reviewers on the motivation of your work, reading this article can help you think about why your research does (or does not) matter.

I would also strongly encourage all computer science Ph.D students to do at least one summer internship (especially in an industrial research lab). You will make new friends and get to work on cool projects. Most importantly, it can truly broaden your herizon and allow you to understand what kind of research matters.

Commonly Used Forms for Home Remodeling/Repairs

2012-06-09T00:49:00-07:00

As a new home owner, we have been going through a lot of home remodeling and repair projects. I found myself often search online for the same set of forms again and again. This post is to record all the forms that I often use and sometimes difficult to found.

1. Simple contract (To be updated)

2. Payment forms (including those for both pregressive payments and final payments)

Baby Cloth Shoe - First Attempt

2012-06-01T00:50:00-07:00

As a busy computer scientist, I don't really get to exercise my inner Martha too much. But I did somehow manage to make my first ever attempt on sewing projects --- baby cloth shoes (as gift for my niece).

Here is the result --- I added the handmade pompom for extra cuteness. :)

The detailed steps can be found from the following link

https://www.michaelmillerfabrics.com/blog/cloth_shoe.pdf

I found the instructions there very easy to follow even for first-timers like myself. However, this is a fairly time consuming project (at least for me). I may do it again, but probably only when I am really bored. :-P

Cat Photography with iPhone 4s

2012-05-19T01:19:00-07:00

It has been a while since Yunyao got her iPhone 4s, which is known for its "superb" camera --- "8MP camera with all-new optics also shoots 1080p HD video". 8MP on a cell phone sounds really impressive, does it? Maybe she finally can leave her big heavy Pentax for a while?

So she tested its performance with her favoriate model: Mayori.

First, let's test it under natural light setting:

"What's up?"

"Count Number of Maps: First Exercises of Conceptual Mathematics in Clojure"

2012-05-16T20:33:00-07:00

As previously mentioned, I am learning category theory, beginning with Lawvere's Conceptual Mathematics book. This is a very elementry book that assumes almost nothing as a background. However, it is still a math book, which requires doing some exercises. Since the book provides no answer to exercises, I decide to make my own and post them here as I did them. Hopefully someone will find them useful.

Since a large part of category theory is constructive, I will try to implement the concepts computationally in order to understand them better. Cateogry theory has been implemented as types in some strong typed languages such as ML and Haskell. I think it would be fun to see how it would look in a dynamic typed language such as Clojure. Even if I could not go very far, at minimum, I will have a mechanical means to check my solutions to the exercises.

The first article of the book deals with the category of sets, and the main topic is about maps between sets. The second article talks about isomorphisms and related concepts. Some exercises (on page 20 and 47) are of the "how many maps are there" variety. Here are some Clojure code I used to calculate the results.

(ns CM.core
(:use clojure.math.combinatorics))

(defn all-maps
"Return a lazy sequence of all the possible maps from a domain to
a codomain"
[domain codomain]
(map #(conj {:domain domain :codomain codomain} [:rule (zipmap %1 %2)])
(repeat domain)
(apply cartesian-product (repeat (count domain) codomain))))

(defn same-mapping-rule?
"Return true if two mapping rules give the same results for a domain"
[domain r1 r2]
(every? identity (map = (map r1 domain) (map r2 domain))))

(defn same-map?
"Return true if two maps are the same"
[f g]
(and (= (:domain f) (:domain g))
(= (:codomain f) (:codomain g))
(same-mapping-rule? (:domain f) (:rule f) (:rule g))))

(defn compose-map
"Return a composed map, also ensure domains and codomains match"
[f g]
(if (= (:codomain g) (:domain f))
{:domain (:domain g), :codomain (:codomain f),
:rule (comp (:rule f) (:rule g))}
(throw (Exception. "Cannot compose, domain does not match codomain"))))

(defn composed?
"Return true if map f and map g compose to map c"
[f g c]
(same-map? c (compose-map f g)))

(defn idempotent?
"Return true if the given map return the same results as when it is
applied twice"
[f]
(composed? f f f))

(defn solutions
"Return a lazy sequence of maps that match the given predicates and go
from the given domain to given codomain"
[pred domain codomain]
(filter pred (all-maps domain codomain)))

(defn idempotent-maps
"Return a lazy sequence of idempotent maps between a domain and itself
as the codomain"
[domain]
(solutions idempotent? domain domain))

(defn choice
"Return a lazy sequence of maps that are applied before the given map to
return the same results as the given composed map, i.e. solution of
choice problem"
[f c]
(solutions #(composed? f % c) (:domain c) (:domain f)))

(defn determination
"Return a lazy sequence of maps that are applied after the given map to
return the same results as the given composed map, i.e. solution of
determination problem"
[g c]
(solutions #(composed? % g c) (:codomain g) (:codomain c)))

(defn section
"Return a lazy sequence of maps that are sections of the given map"
[f]
(choice f {:domain (:codomain f) :codomain (:codomain f) :rule identity}))

(defn retraction
"Return a lazy sequence of maps that are retractions of the given map"
[f]
(determination f {:domain (:domain f) :codomain (:domain f) :rule identity}))

(defn composed-solution-pairs
"Return a lazy sequence of pairs of maps that compose to a given map, with
the given shared domain in between"
[domain c]
(filter
#(composed? (first %) (last %) c)
(for [g (all-maps (:domain c) domain)
f (all-maps domain (:codomain c))]
[f g])))

(defn retraction-section-pairs
"Return a lazy sequence of pairs of maps r and s, where r goes from domain
X to domain A, s goes from A to X, and r of s is the same as the identity
map on A."
[A X]
(composed-solution-pairs X {:domain A :codomain A :rule identity}))

(def A #{"John" "Mary" "Sam"})
(def B #{"eggs" "coffee"})
(def U #{:b :p :q :r :s})
(def D #{0 1})

(def one-a {:domain A :codomain A :rule identity})
(def one-b {:domain B :codomain B :rule identity})
(def one-d {:domain D :codomain D :rule identity})

(def g {:domain U :codomain D :rule {:b 0 :p 0 :q 0 :r 1 :s 1}})
(def f {:domain D :codomain U :rule {0 :b 1 :r}})
(def u {:domain D :codomain U :rule {0 :b 1 :b}})

In this code, maps in category theory are simply implemented as Clojure's map data structure. So a map in category theory will have three keys in the implementation, a :domain, a :codomain, and a :rule. The first two are sets, and the last one is a function, which defines the actual mapping rule from domain to codomain.

Article 1, Exercise 2. How many different maps $f$ are there with domain $A$ and codomain $B$?

Domain $A$ has 3 elements "John", "Mary" and "Sam", codomain $B$ has 2 elements "eggs" and "coffee". To find the answer, evaluate this in REPL:

CM.core=> (count (all-maps A B))

For our domain of 3 elements and codomain of 2 elements, the number of maps is 8, or $2^3$. If we try some other domains (Exercise 3, 4, 5), we will soon discover that the answer is always $$n^m$$ where $n$ and $m$ is the size of codomain and domain, respectively. The reason is simple: each and every element of the domain can pick any one of the elment of codomain as the target. So the first element in the domain has $n$ choices of target, the second can pick $n$ choices as well, and so on, doing this $m$ times, and multipling them all up gives the answer.

Article 1, Exercise 6. How many different maps $A \stackrel{f}{\longrightarrow} A$ satisfy $f \circ f = f$?

Basically, this is asking how many maps are there that composes with itself to get the same results as itself. Such maps are called idempotent maps. The code to find the answer:

CM.core=> (count (idempotent-maps A))

Let's try Exercise 7, the number of idempotent maps for $B$:

CM.core=> (count (idempotent-maps B))

How about a domain with 4, 5 or 6 elments?

CM.core=> (count (idempotent-maps #{1 2 3 4}))

CM.core=> (count (idempotent-maps #{1 2 3 4 5}))

196

CM.core=> (count (idempotent-maps #{1 2 3 4 5 6}))

1057

Hmm, 3, 10, 41, 196, 1057, ... what is the regularity here? It is not obvious. Let's examine the details of the first two maps and look at their rules:

CM.core=> (map :rule (idempotent-maps B))

({"eggs" "coffee", "coffee" "coffee"} {"eggs" "eggs", "coffee" "coffee"} {"eggs" "eggs", "coffee" "eggs"})

CM.core=> (map :rule (idempotent-maps A))

({"Sam" "John", "Mary" "John", "John" "John"} {"Sam" "Sam", "Mary" "John", "John" "John"} {"Sam" "John", "Mary" "Mary", "John" "John"} {"Sam" "Mary", "Mary" "Mary", "John" "John"} {"Sam" "Sam", "Mary" "Mary", "John" "John"} {"Sam" "Sam", "Mary" "Sam", "John" "John"} {"Sam" "Mary", "Mary" "Mary", "John" "Mary"} {"Sam" "Sam", "Mary" "Mary", "John" "Mary"} {"Sam" "Sam", "Mary" "Mary", "John" "Sam"} {"Sam" "Sam", "Mary" "Sam", "John" "Sam"})

The regularity seems to be this: either an element must map to itself, or it must map to an elment that maps to itself. For example, for domain $B$, if "eggs" maps to itself, "coffee" must either map to "coffee" or to "eggs". Cross mappings are not allowed. So "eggs" maps to "coffee" and "coffee" maps to "eggs" is illegal.

Given this regularity, let's work out a formula for the number of idempotent maps. For a 2 element domain, there are two cases: both elements map to themselves, or both map to one element, so the total is ${2 \choose 2} + {2 \choose 1} = 3$; For a 3 element domain, there are three cases: all map to themselves, two map to themselves and the third maps to one of them, or all three map to one, total is ${3 \choose 3} + {3 \choose 2}{2^1} + {3 \choose 1} = 1+ 3\times2 + 3 = 10$; For 4 element domain, four cases: all map to themselves, three map to themselves and the fouth to one of three, two map to themselves and remaining two map to those two, or all map to one: ${4 \choose 4} + {4 \choose 3}{3^1} + {4 \choose 2}{2^2} + {4 \choose 1} = 1 + 4\times3 + 6\times4 + 4 = 41$; and so on... A general formula for the number of idempotent maps emerges, it is $$\sum\limits_{k=0}^n {n \choose k}k^{n-k}$$ Where $n$ is the size of the domain. Notice that we used the results of the previous exercise in the derivation: the number of maps is $|codomain|^{|domain|}$.

Article 1, Exercise 8. Can you find a pair of maps $A \stackrel{f}{\longrightarrow} B \stackrel{g}{\longrightarrow} A$ for which $g \circ f = 1_A$?

OK, this is asking if we can find a pair of maps that compose to an identity map of domain $A$, with the map $B$ in between. Let's see:

CM.core=> (count (composed-solution-pairs B one-a))

No such map pair exists, so it is not possible to go through $B$ back to $A$. What about going through $A$ itself?

CM.core=> (count (composed-solution-pairs A one-a))

There are 6 such map pairs. It is the same as going through another 3 element domain:

CM.core=> (count (composed-solution-pairs #{1 2 3} one-a))

So when the domains have the same number of elements, it is possible for the map compositions to go through them back and forth. Let's look at the details of these map pairs:

CM.core=> (composed-solution-pairs #{1 2 3} one-a)

([{:rule {3 "Sam", 2 "Mary", 1 "John"}, :domain #{1 2 3}, :codomain #{"John" "Mary" "Sam"}} {:rule {"Sam" 3, "Mary" 2, "John" 1}, :domain #{"John" "Mary" "Sam"}, :codomain #{1 2 3}}] [{:rule {3 "Mary", 2 "Sam", 1 "John"}, :domain #{1 2 3}, :codomain #{"John" "Mary" "Sam"}} {:rule {"Sam" 2, "Mary" 3, "John" 1}, :domain #{"John" "Mary" "Sam"}, :codomain #{1 2 3}}] [{:rule {3 "Sam", 2 "John", 1 "Mary"}, :domain #{1 2 3}, :codomain #{"John" "Mary" "Sam"}} {:rule {"Sam" 3, "Mary" 1, "John" 2}, :domain #{"John" "Mary" "Sam"}, :codomain #{1 2 3}}] [{:rule {3 "Mary", 2 "John", 1 "Sam"}, :domain #{1 2 3}, :codomain #{"John" "Mary" "Sam"}} {:rule {"Sam" 1, "Mary" 3, "John" 2}, :domain #{"John" "Mary" "Sam"}, :codomain #{1 2 3}}] [{:rule {3 "John", 2 "Sam", 1 "Mary"}, :domain #{1 2 3}, :codomain #{"John" "Mary" "Sam"}} {:rule {"Sam" 2, "Mary" 1, "John" 3}, :domain #{"John" "Mary" "Sam"}, :codomain #{1 2 3}}] [{:rule {3 "John", 2 "Mary", 1 "Sam"}, :domain #{1 2 3}, :codomain #{"John" "Mary" "Sam"}} {:rule {"Sam" 1, "Mary" 2, "John" 3}, :domain #{"John" "Mary" "Sam"}, :codomain #{1 2 3}}])

Obviously, for each pair, the two map rules are simply the reverse of the another, i.e. flipping the arrows around. If a map has an inverse, it is unique. These maps are called isomorphic, bijective, or one-to-one and onto.

In fact, when the map in the middle has larger size than $A$, there may also be map pairs that compose to the identity of $A$.

CM.core=> (count (composed-solution-pairs #{1 2 3 4} one-a))

In such pairs, each map is called retraction and section to each other. Let's calculate the number of retractions and sections.

Article 2, Exercise 5 (1) Given map $g$ (see code for its definition), how many maps $f$ are there with $g \circ f = 1_{{0, 1}}$?

This is asking the number of sections of the map $g$, which has a 5 element domain $U$, and a 2 element codomain $D$. Three of the elments of $U$, b, p, q, map to 0 in $D$; two elements of $U$, r and s, map to 1 in $D$. The answer can be found by:

CM.core=> (count (section g))

Basically, each section $f$ must choose two elements in $U$ to map 0 and 1 to, such that $g$ can map the results back to form an identity map on {0, 1}. For element 0, $f$ can choose one of b, p or q to map to; for element 1, $f$ can choose one of r and s. Therefore, the number of possible $f$ is $2 \times 3 = 6$.

Obviously, not all maps have sections.

CM.core=> (count (section f))

To have sections, the map must have a domain size larger than or equal to the codomain size. In addition, each elment of the codomain must be mapped to. Such map property is called surjective or onto. The general formula for the number of sections for map $g$ is therefore $$\prod_{i=1}^{n}m_i$$ where $n$ is the size of the codomain of $g$, and $m_i$ is the number of elements in the domain of $g$ that map to the $i$th element of the codomain.

Article 2, Exercise 5 (2) Choose a particular such $f$ (see code for its definition), how many maps $g$ satisfy $g \circ f = 1_{{0, 1}}$?

Given a chosen $f$, this question is asking its number of retractions. The answer is:

CM.core=> (count (retraction f))

For given $f$, 0 and 1 each maps to its own element in $U$, its retraction only need to flip the arrows to point back to 0 and 1, the remaining three element in $U$ can freely choose any of 0 and 1 to map to, so the number of retractions is the same as the total number of maps from a 3 element domain to a 2 element codomain, $2^3$. The general formula is $$n^{m-n}$$ where $n$ is the size of domain of $f$, and $m$ is its codomain size. To have retraction, a map must have a domain smaller or equal to the size of its codomain. In addition, it must be a one-to-one mapping, also called injective mapping.

Number of section-retraction pairs On page 117 of the book, the above formula for the number of retractions and sections of a given map are given, but it also says that the formula for the number of pairs of section/retraction in term of $m$ and $n$ is rather complicated. As it turns out, it is simple to derive a formula that is not complicated at all. First, each element of the smaller domain (size $n$) must map injectively to the lager domain (size $m$), the number of possibilities is just the number of permutation of choosing $n$ out of $m$. Then the remaining $m-n$ in the larger domain can freely choose any of the $n$ to map back to. Finally, we time up the two terms to arrive at $$\frac{m!}{(m-n)!}n^{m-n}$$ where $n$ is the size of domain $A$, m is the size of the domain $X$, $n \le m$, and $A \stackrel{s}{\longrightarrow} X \stackrel{r}{\longrightarrow} A$ satisfy $r \circ s = 1_A$.

The formula seem to be correct as verified by the code:

CM.core=> (count (retraction-section-pairs #{1 2} #{:a}))

CM.core=>(count (retraction-section-pairs #{1} #{:a}))

CM.core=> (count (retraction-section-pairs #{1 2} #{:a :b}))

CM.core=> (count (retraction-section-pairs #{1 2} #{:a :b :c}))

CM.core=> (count (retraction-section-pairs #{1 2} #{:a :b :c :d :e}))

160

CM.core=> (count (retraction-section-pairs #{1 2 3} #{:a :b :c}))

CM.core=> (count (retraction-section-pairs #{1 2 3} #{:a :b :c :d}))

CM.core=> (count (retraction-section-pairs #{1 2 3 4} #{:a :b :c :d}))

CM.core=> (count (retraction-section-pairs #{1 2 3 4} #{:a :b :c :d :e}))

480

CM.core=> (count (retraction-section-pairs #{1 2 3 4 5} #{:a :b :c :d :e}))

120

CM.core=> (count (retraction-section-pairs #{1 2 3 4} #{:a :b :c :d :e :f}))

5760

CM.core=> (count (retraction-section-pairs #{1 2 3 4 5} #{:a :b :c :d :e :f}))

3600

Don't try the last two function calls, as they will run a long long time.

Install Vim with Homebrew Python on OS X

2012-05-16T04:50:00-07:00

For people that need full features of vim, the default installation of vim on Mac OS X is definitely not enough. For example, I need to use vim to post to this blog, which requires a version of vim with python support. I also prefer terminal version of vim to the GUI version, so MacVim is less desirable.

One way to get what I want is to compile a version of vim with homebrew. Homebrew does not officially have a vim fomula, because that would be a duplicate of the system version. Fortunately, we can grab an unofficial formula at https://raw.github.com/Homebrew/homebrew-dupes/master/vim.rb

Also, you may want to install the latest homebrew python with the framework option:

brew install python --framework

In order for vim to use the homebrew version of python (2.7.3) instead of the system one (an old 2.6), we need to change a line and add a line in the downloaded formula (vim.rb) file:

...  
"--enable-pythoninterp=dynamic",  
"--with-python-config-dir=/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config",  
...

Now install vim with the formula:

brew install ./vim.rb

To check vim is using the correct version of python. Issue command in vim:

:python import sys; print sys.version

If we get something like this:

2.7.3 (default, May 15 2012, 20:51:34)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
Press ENTER or type command to continue

we are done!

Start learning category theory

2012-04-30T07:48:00-07:00

Perhaps due to my rather small brain (literally), I dislike remembering tedious details. When in elementary school, I hated reciting classic Chinese poems, but liked composing my own :-). In high school, I hated chemistry but loved physics, because one could do everything based on a few principles in physics, whereas chemistry was all about memorization. Last year, I was chatting with a colleage of mine who had a Ph.D. degree in chemistry from Harvard. He said he's good at it because he could find patterns in all the tedious details and summerized them in his own head, so he didn't have to remember them all. So I said why not write those patterns down so others can benefit, and he didn't seem to like that idea. Anyway, let's go back to the main topic.

I did not do very well in math classes in China, perhaps because Chinese math education focused too much on problem solving techniques, which were basically a matter of remembering all the test coping tricks. But I do like math, because I find it elegant and profound. A mathematical understanding of the world seems always the most economical one. So I want to learn more.

To consistently apply my preferred model of starting from the most fundamental ideas, I decide to learn category theory. It might sound strange as even many professional mathematicians consider this branch of mathematics "abstract nonsense", but I think it might fit my aptitude. Who knows. In any case, I am following John Baez's book recommendation as a guide, and will be updating my progress here from time to time, when I have something interesting to write about.

UPDATE: Baez, among others, said they benefited from Goldbatt's book, which I myself find very readable as well. However, this MacLarty's article convinced me that Goldbatt's book might give the wrong idea about category theory. So I will stick with Lawvere's books.

"It's official: Tom is a Nebelung!"

2012-03-24T23:07:00-07:00

When Tom (唐唐）was adopted from the animal shelter, his papers listed him as an "American Domestic Long Hair". As new cat parents, we were eager to find out his true breed. We searched and searched online, and identified him as a Chartreux, based on his fur color and personality. We knew that Chartreux is a short hair cat but Tom has long hair, but we thought Tom must not have been a pure breed. We always joked about Tom's purpoted French origin and attributed his love of bread and butter to his French ancestry.

Yesterday we took the cats to get their teeth cleaned. Sitting in the vet's waiting room, I browsed a cat breed book, and stumpled upon Tom's true breed. He's a Nebelung! Let's see. At the left is a picture of a Chartreux, and at right a Nebelung:


Chartreux	Nebelung

Obviously, Chartreux has short hair, orange eyes, rounder head and body, while Nebelung has long hair, green/yellowish eyes and leaner head and body. No doubt, Tom is a Nebelung. Many of the google images of Nebelung cats can just be passed as Tom's own photos.

Here's Tom:

So Nebelung is a rare breed created in America during 1980s from Russian Blue. This behavior characteristics of Nebelung reads like Tom's accurate description:

Nebelung cats are lively, playful, affectionate, good-natured, and intelligent. The cat's mild-mannered state and personality may not always reflect the relatively high level of intelligence often found in the breed. In spite of the fact that it is an active cat, it can live very well indoors. Nebelungs prefers their own family and often keep a distance from strangers. They tend to bond with a select few humans and stay loving and devoted throughout their lives. It is, however, a cat that gladly accepts company of its people or of another cat. Nebelungs are very good communicators and will remind their owners of problems. This happens often since Nebelungs can be very picky about things such as litter cleanliness and food type. Many owners of the Nebelung cat say that they can act more like a dog at times, being extremely loyal to their owner and family. It enjoys sitting in a lap and being petted and will follow its favorite person devotedly from room to room.

Now we know we are not crazy calling Tom a "Mommy's boy" and a "dog cat", because he is a Nebelung!

Are you Han? My paternal ancestry - 12 Marker Results

2012-03-03T01:04:00-08:00

As mentioned previously, I swabbed my cheeks and mailed my Y-DNA test specimen to FamilyTreeDNA a few months ago. Now the first part of the results, consists of twelve markers, are known. These are standard Y-STR results. Before showing the actual numbers, let me briefly explain.

What I ordered was a Y-DNA test, which is only applicable for men, since women do not have Y-chromosome. Y-chromosome passes from fathers to sons, so it can be used to track male ancestry. Chromosomes consist of a large number of nucleotides, which have four basic types, designated by letter A, C, G, and T. At certain locations of the Y chromosome (these locations are usually labeled by names such such "DYS393"), there are repeated sequence of short letter patterns, such as "GTT", called Short Tandem Repeats (STR). For example, at location DYS426, a person may have 7 to 18 repeats of the DNA sequence “GTT”, depending on the individual. In the FamilyTreeDNA database, 12 repeats at DYS426 is the most common, which would look like this: ...TGTGTTGTTGTTGTTGTTGTTGTTGTTGTTGTTGTTGTTGAC... By comparing the number of such STRs at different locations, we can estimate how genetically close two individuals are. And we call the number of STRs at each of these locations a marker.

Because STRs mutate more easily, two individuals with a similar Y-STR haplotype may not necessarily share a similar ancestry. To really confirm one's haplogroup type, Single Nucleotide Polymorphism (SNP) test is needed, which examines single nucleotides at specific locations on the Y chromosome. Since these SNPs are very rare, people sharing the same SNPs almost certainly have the same ancestor many generations ago. In summary, STR test is good for finding recent relatives, while SNP can confirm one's ancient ancestry (i.e. haplogroup).

My Results

The table below shows the results of 12 markers for my Y-DNA test.

Huahai's 12 Marker Results
Marker	Value
DYS393	12
DYS390	24
DYS19**	15
DYS391	10
DYS385a	12
DYS385b	20
DYS426	11
DYS388	12
DYS439	12
DYS389I	13
DYS392	13
DYS389II***	31

I have yet to figure out that those stars mean.

OK, the table looks nice, but what does it mean? Well, given this data, we can do at least two things: the first is to find one's remote genetic cousins. Another thing is to predict one's haplotype.

My Match

I searched the FamilyTreeDNA's database to find people with test results that match mine. BTW, this search functionality is also available at ysearch.org. Guess what? I found one person with an exact match! And I know his name! I have not contacted that person yet, because I want to wait for more results to be sure. That person did a 25 marker test, so I will wait for my 25 marker results.

However, I have not resisted the temptation to investigate a bit about his background. Curiously, his oldest known ancestor is from Philippine, dating back to late 19th century. On the other hand, according to our family records, my father's line has lived in Sichuan province in southwestern China since Ming Dynasty, way before 19th century!

So what's the connection here? There seems to be so much room for speculation. One possibility is that this Filipino family was originally migrated from China to Philippine a few hundred years ago. Given this family rather scholastic background (complete with a prominent medical scientist in the family that has his own Wikipedia page), this hypothesis doesn't sound too far off, as Chinese people are known to keep their scholastic habit even overseas. Another evidence is the similarity of surnames. This family's surname is pronounced vaguely similar to mine, so it could be a local adaptation of the original Chinese surname.

Another possibility, is of course that my family was actually migrated to Sichuan from the pacific coast (maybe even the islands) somewhere during the Ming Dynasty. This would not sound too far off either, since my family records does say that we migrated to Sichuan during Ming Dynasty. However, it is vague on the origin of the migration.

In any case, this seems to be getting very interesting. I am looking forward to seeing more test results.

My Haplotype

People have developed statistical software to predict haplogroup from STR test results. For example, Whit Athey's predictor is popular for people of European or Mideastern origins, but it doesn't work for Asians perhaps due to a lack of data. On the other hand, FamilyTreeDNA does prediction for all customers. Before showing the prediction for my haplogroup, let's recap my own prediction in the previous post: my hypothesis is that I have 20 percent chance to be in haplogroup K*, 20 percent chance in haplogroup O2, 30 percent in O1 and O3, respectively.

Well, how did I do?

Pretty good, in fact. Basically, before seeing the results, I predicted that I have 20+30+30=80 percent chance to be in the haplogroup O. It turns out that FamilyTreeDNA's prediction is just that: I am in the haplogroup of O!

Further results should get me closer to the exact haplogroup within O. But to have some fun, let's predict again which sub-group I am in. Given that one exact match I have in the database is someone originally from Philippine, the chance of my being in O1 should be significantly increased, whereas the chance of being O2 should be diminished, down to zero. Therefore, my hypothesis is the following:

P(O1) = 0.7

P(O3) = 0.3

Let's wait and see...

UPDATE (3/7/2012): I found another haplogroup predictor, Vadim Urasin's YPredictor, which can handle Eastern Asian results. The prediction for my 12 marker results is the following:

N Haplogroup Probability
1 O3-M122 77%

What a surprise! So I am a typical Han Chinese after all.

On the other hand, the accuracy of this predictor may not be too high for Eastern Asians due to a lack of data. Anyway, looking forward to results from more markers to confirm...

UPDATE (3/7/2013): I paid for the deep-clique analysis to get to the bottom of this, and the results confirmed Vadim Urasin's prediction, that I am indeed O3-M122. So in this ense, I am a typical Han Chinese male. The only thing atypical is that I do not belong to any of the more popular branches under O3-M122. This result seems to explain a lot, since I am in one of those rare branches under O3.

Review for Canine Cat Scratcher

2012-01-02T06:18:00-08:00

Updated on 09/10/2013: It's now only 295 USD, with 100 USD discount!

As a cat and modern design lover, I am always looking for cat items pleasant for both the cats and my aesthetic standard. I had my eye on the canine scratcher for a while, but its over 1000 USD price tag made it out of reach. However, recently, I found that it now costs "only" 395 USD. I pulled the trigger w/o too much hestitation since I need to buy the cats a holiday present anyway. However, I was still a little nervious at the purchase, as it would be a huge bummer if the cats don't like it. But I really love the design and can totally see how it fit with the rest of the house w/o screaming "cat furniture".

It took over 3 weeks to receive the scratcher (somehow I didn't even get the confirmation email and it took an email from me to learn about the status of my order). But I was very pleased at the arrival. It was very well packaged and arrived without any damage. It is just the right size as a standalone decoration. Most importantly, the cats love it.

After some fighting, it has now official become Mayori's favorite ride. Highly recommended!

Are you Han? My paternal ancestry - Hypothesis

2011-12-28T07:02:00-08:00

I was brought up as a Han Chinese and I myself have never doubted that identity. However, I do know that I look slightly different from people around me when I grew up. Darker skin, slightly curly hairs, deeper set eyes and other distinctive facial features. Put it simply, my face does not look like an average Han male face. However, nobody really said anything about my appearance except that my old brother occasionally joked about it (he looks like a normal Han Chinese by the way). After all, I look just like my father. Besides, there are plenty of male relatives in my father's native village (near Chongqing) who look just like me.

It took a foreigner to first question my ethnicity. When I was in graduate school in Beijing, there was an exchange student in my group, an African American girl, who one day took me by surprise by asking: "Are you Han?" I instinctively answered "Yes". She didn't say anything more and I didn't either, but I knew why she asked.

When I came to US, the situation got worse. When I went with Yunyao to visit her college friends, they always talked English with me first, then were totally startled by my almost perfect Mandarin. Hispanic people in the states always wanted to talk Spanish with me then were surprised by my inability to speak their language. Later I learned one Spanish phrase or two. When we traveled in Latin American, people in the restaurants always insisted on correcting my poor Spanish pronunciation while ignoring Yunyao's arguably worse performance.

Since all the female members of my family look like normal Han people, Yunyao became curious about my paternal ancestry and suggested me to take a DNA test to find out. At first, I dismissed this seemly outrageous idea. A few months ago, I became curious myself when I learned a few things about the make-up of Han Chinese Y-DNA haplotypes. Below is a chart I made using the data from this Nature study on 1289 Han males in year 2004.

O3*, O3d and O3e all belong to the group O3. It is interesting to see that O3 takes up about 55 percentage of the Han populations.

Since people always talk about the differences between southern and northern Chinese, a chart (using the same data set) comparing the two follows:
Clearly, Northern and Southern Han Chinese males have similar genetic distributions, so the alleaged north-south difference is not that significant, at least in term of male lineage.

Now, the big question, what haplogroup do I belong to? To find out, one has to take out the wallet 😛. Luckily, Yunyao bought this FamilyTreeDNA test for me as a birthday gift. Here is the package I just received in the mail:

There are two swabs and two specimen tubes in the package. Following the instructions, I scraped my cheek for a full minute, put the swab in the tube, and put everything into the envelop provided, and mailed the package back.

While waiting for the results, let's make a prediction about my haplogroup. To do that, let's look at the haplogroup distribution of the province where my father and I came from (using the same data set as above):

The number of Sichuan Han people tested in the Nature 2004 study is only 65, so this chart may not be very accurate. In the absence of better prior information, let's work with this.

There are 5 percent of C*among Sichuan Han people. A major part of these are likily the decendents of the Mongol invaders. Since I am not Mongol looking, I rate the probability of my being C* to be zero, P(C*)=0.

The two percent of D1 in Sichuan Han people are likely related to a Tibetan origin. According to my family history, this is again unlikely my paternal lineage, P(D1)=0.

There are surprising large percentage of K* shown in the data. K is a very old lineage appearing around the Pacific everywhere, and is particularly common in the Australian islands. I think the chance of my being in this group is higher than normal. Of course, it could be that the study didn't not differentiate some of the down-stream haplogroups of K, such as LT, M, N and S. In any case, I rate the probability of my being one of these K people higher than 16 percent, P(K*, LT, M, N, or S)=0.2.

Similar to the national level, O3 is the most common among Sichuan Han people, taking up 57 percent. Since I am not normal Han looking, I will down rate the probability of my being O3 from 0.57 to 0.3, P(O3)=0.3. I cannot down rate O3 too much though, as my blood brothers do look like normal Han.

The eight percent of O1 in Sichuan Han people are likely the results of migration from other parts of southern China, as this is a common gene type in southeastern Asia. Given our family history, the probability of my being O1 should be significantly higher, P(O1)=0.3.

O2 accounts for 13 percent of Sichuan Han people. O2a is commonly seen in the tribal people of southern Asia. I rate the probability of my being O2 higher than normal: P(O2)=0.2.

To recap, my probablistic hypothesis regarding my own Y-DNA haplogroup is the following:

P(K*, LT, M, N, or S)=0.2

P(O3)=0.3

P(O1)=0.3

P(O2)=0.2

Let's wait for the moment of truth...

Best Chocolate Places in Manhattan

2011-11-13T21:02:00-08:00

I recently stayed in Manhattan for a few days during a business trip. My colleague and I were looking for places to buy some special chocolate to bring back home. We asked a few locals New Yorker as well as consulted the web for recommendations. Here are the list of stores we collected.

La Maison Du Chocola (store website)
Vosges (store website)
Jacques Torres (store website)

We only have 3 hours to spend in the city. So we went to La Maison Du Chocola store in the Rockefelle Center, the store closest to our hotel.

Since there is not free sampling, I was not very sure that I would like to spend $100 on one pound of chocolate. I bought two small boxes with 6 pieces of chocolate each. Obviously, I am not the only one who is not sure: they even have tiny small packages with only two pieces of chocolate (for $8), accompanied with a pretty gift bag.

When I gave the gift to Huahai. He absolutely loves the chocolate. So it is money well spent and I plan to buy more for him when I go to NYC next time.

Review for "Virtual Cat Toys HD Catnip Collection"

2011-11-06T19:46:00-08:00

As many of friends, I recently acquired a HP Touchpad largely due to its low price. I don't really need a tablet, as I am sitting in front of computes almost all the time. My original plan was to turn it into an interactive cat toy, at least sometimes, so I can keep the two cats entertained while I am working.

Unfortunately, I could only find one game designed for the cats "The Catnip Collection". According to its website, it "is a variety of toys that will amuse and challenge your cat and its human".

The reviews seem pretty decent (4 out 5).

https://developer.palm.com/appredirect/?packageid=com.wtflolomgbbq.haxe…

Even though it is not free anymore, I bited the bullet and bought the app. Unfortunately, the cats are not even remotely interested! I think that the problem is the objects (insects, balls, etc) are constantly moving instead of stopping and moving randomly. So the cats get bored after 5 seconds.

Hopefully the app developer will see my post and consider improve the game further. For the smart cats who play with human all the time, we need something that is really challenging!

Planning a Dream Trip to Italy the Easy Way

2011-10-16T22:13:00-07:00

Huahai and I have just returned from our first trip together to Italy. The trip itself was wonderful and unforgeable. However, it usually took a lot of planning for such an extensive trip, unless you hire someone else to do it for you or join a tour group.

We usually enjoy traveling at our own pace and explore the area; therefore, we (basically myself) always do our own travel planning. Unfortunately, as a very busy professional, I do not really have much time to do the planning. (Can you believe someone have actually spent half a year to plan a two-week's trip to Italy and produce a travel plan with 90 pages?)

Following my usual practice, I have spent a lot of time initially on online travel forums initially. I also got the aforementioned 90-page's travel plan and a shorter one from a friend. Then I have brought Rick Steven's book. Eventually, I have managed to come up with a travel plan with only two pages that working out pretty well. Here are some tips that I have summarized based on my experience with this trip. I wish that I have known them first before I started the planning. It would have made my life much easier without compromising the quality of the travel plan.

1. Buy at least one good travel book.

For European trips, I highly recommend Rick Steven's book, which contains very detailed information for each of the places. Then you can bring the book with you and do some of the detailed planning such as where to have lunch on the go. I bought this book only a couple of weeks before our trip. I wish that I have bought it before I even started the planning. It would have saved me a lot of time reading online postings.

2. Refer to travel packages for possible travel itinery.

While I do enjoy the freedom of traveling at our own pace, studying the sample travel packages listed at the travel agency's websites is a very efficient way to come up with a travel plan that covers the major must-seen sites. I would add one extra day for each stop, as the tour groups tend to travel at much faster pace than I would like. Rick Steven's websites also provide very good example routes for each city with free mp3 audo downloadable. Another good place to get some inspiration is Yahoo! trip planner, where you can view other people travel plans.

3. Book hotels well in-advance.

A good travel book typically contains some recommendation for accommodations. But the list is usually very short. I found Tripadvisor to be a great place to find accommodations with good value without broking the bank. However, top ones can sold out fast. Therefore I recommend booking the hotels as soon as the trip schedule is set. Of course, don't forget to use Kayak.com to find out how to get the best price. I booked the hotels about two months ahead of time, but many of the smaller nice budget hotels have already sold out.

4. Reserve museums in-advance or join private tours.

Check the travel book or online forums to find out whether any of the sites on the travel plan requires reservation and make the reservations ahead of time. Otherwise, you may find yourself stuck in a really long line for hours or even worse end up have to miss the must-seen sites. Joining private tours typically helps avoiding the long line but can be expensive. We only did the private tours when we were able to see things that we could not see otherwise (e.g. underground part of Colloseum is only accessible by joining one of the private tours). Many of the places can be reserved online. In addition, many hotels can help you make the reservations for free --- just send requests via emails with desired time and date.

5. Arrange only one or two must-do items per day, preferably in the morning.

I feel that vacation is a time to relax, not a time to check of boxes of to-do items on the travel plan. Therefore, it is better to do only one or two must-do sight-seeings in the morning and then simply add other items as you go depending on the mood. With a good travel planning book, it is easy to do such last-minute planning without stressing out yourself in planning or following the plan.

6. Arrange some resting days.

As a first-timer to Italy, I was surprised how much walking I need to do everyday. It is true that most of the things are either within walking-distance or reachable via public transportation plus walking. However, all these walking adds up! According to my pedometer, we walked over 10 miles in one single day just visiting museums! Unless you would like to walk miles and miles everyday, it would be better to arrange some days with activity that requires not too much walking so your feet can recover. I am really glad that between our visits to Rome and Florence, we stayed in Sorrento and did a few day trips involving bus tour, boat tour, a little bit hiking, and swimming. These resting days give us some space to enjoy Italy from a completely perspective while giving our feet a break.

Best CSV Reader/Writer?

2011-10-11T21:36:00-07:00

My colleague has highly recommended to me the following CSV reader/writer: https://supercsv.sourceforge.net/

I am yet to test it out, so I will need to come back provide my review. But just put a pointer here, since this tool is not realy easy to find without the right keywords.

"Quick Fix: Windows XP Crashes with Blue Screen with Video Content"

2011-09-22T00:06:00-07:00

For some reason, my main working machine, a Windows XP desktop, recently started to crash a lot followed by the blue screen of death. The error message output at the blue screen does not seem to be helpful.

***STOP: 0x0000008E (0xE0000001, 0xBA490925, 0xA880D820, 0x00000000)
***watchdog.sys - Address BA490925, base at BA90000, Date stamp 480254ab

After a few crashes, I realized that the machine crashes whenever I play any video in the web browser ( I am using FireFox). Initially, I didn't bother to look into this problem and simply attempt to solve the problem by avoid playing any video, since I don't really intent to watch any video during my work hours and would rather spend my time doing really work instead of fixing the machine. However, this didn't really help since so many sites now a days include video content .

So today, when the machine crashed again, I decided to fix this issue once and for all. Luckily, one Google search using the error message turned up quite a few answers. Most of the answers tent to be too complicated than what I wanted and involving checking for malware, virus, etc, which is not really my concern. Luckily, I found a very simply solution here.

Due to changes to the link, the exact solution from the above link no longer work. But the basic idea is to go to any site with flash content, and then right click on the flash content, select "setting" from the pop-up menu and uncheck "Enable hardware acceleration".

For example, you can go to https://www.adobe.com/products/flashplatformruntimes/gallery/ or any Youtube page. Just make sure that you don't play the video before you make the change. Otherwise, you will see the blue screen again.

Netflix Is Splitting Into Two!?

2011-09-19T23:20:00-07:00

It has turned out that Netflix has abandoned the plan, finally.

When Huahai told me the news late last night that Netflix is splitting into two, my first reaction was that he must be reading from Onion! After all, based all what have learned from the business school classes and information economics classes, it makes perfect sense to provide bundled services, especially when it comes to information product.

However, the news is definitely true, as further confirmed by the following letter that I received this morning.

I don't know about you. But like many other loyal NetFlix subscribers, I love the convenience of their streaming service and almost use it exclusively even when I have bundled unlimited DVD service. In fact, I don't even watch cabled TV anymore except for the morning news. However, when I was informed that I have to pay 60% to get both services, I dumped the DVD service without hesitation. After all, I was only watching at most one DVD per week. However, this price hack does have hurt my feeling --- it seems that they don't care about their customizes anymore when they now dominate the market.

Nevertheless, the above news has shocked me. Frankly, I don't think that this move makes much sense --- why lose the establish name and give away the convenience to the customizes to be able to combine both services and maintain one single account? It also goes again the economic of scale. It all seems to me to be a very costly move.

However, I am very excited at the news. It looks like a big move that will be written into business school text books in the near future. I am really looking forward to seeing the outcome a couple of years from now on.

More on the move can be found on their blog.

Easy Tuna Birthday Cake for the Cats

2011-09-18T23:25:00-07:00

I was looking for buying birthday cakes for the cats' birthday. Surprisingly, although it is very easy to find dog bakeries, there seems no existence of a single cat bakery. Luckily, a single Google search online brought up many birthday cake recipes for the cats. I picked one simple recipe with my cats' favorite food --- tuna.

So here is the tuna birthday cake for the cats, decorated with shrimp with catnip garnish.

It looks very nice, but most importantly, it was an instant hit. The cats jumped on to the dinning table as soon as I put the cakes on it.

Mayori started with her favorite --- shrimp --- first.

Tom enjoyed the cake itself too.

Recipe (modified based on an online recipe I found, but I could not find the original link anymore):

1. Turn the oven on 350 degree

2. A can of tuna or salmon in water + 2 eggs + a little bit cheese + a little bit of flour

3. Mix all the ingredients and put into a muffin pan.

4. Bake for 15 minutes

5. Garnish with a cooked shrimp (the easiest way is to put one or two shrimp into a cup of water in a tea cup. Microwave for 1 -2 minutes).

Of course, I always pick something nice from this list as their birthday gift!

蛋糕的做法如下：

先把烤箱调到350度。

然后将tuna／salmon 罐头一个, 鸡蛋两个，少量cheese，少量面粉，搅拌好以后倒到
烤muffin的模子里。

放烤箱内烤15分钟左右, 上面再放上煮好的虾一只就好了。

Resources to Get Rid of Lawn the Frugal Way (Phrase I)

2011-09-05T01:39:00-07:00

We are getting rid of our front lawn --- to save water and money!

A landscape designer helped us did the design to convert our front yard from an old deteriorating classic English garden into a vibrant modern garden full of native plants.

Given our shoe-string budget, we plan to do most of the work ourselves. However, since both of us work full time, it will be a while before we can actually start the project, but being a person who loves to plan, I have started collecting resources to do the project frugal way already!

Here are the resources I have collected so far based on the main steps for the landscape projects:

Step 1. Demolition

We need to get rid of some plants and get rid of the lawn. It would cost a fortune to hire professional to do all the work, especially given the size of our front yard. So my current plan is to wait for the rain and then get rid of the lawn via sheet mulch.

This video on YouTube explains in details how to do sheet multch. This websites added some more way to amend soil during sheet multch.

Here is a link describing alterative ways to get rid of the lawn.

There are several ways to get cheap compost in the Bay area. The ones I am considering are

Guadalupe Landfill “Valley’s Pride” Compost
999 Guadalupe Mines Road
San Jose CA 95120
408 268-1694
Hours: Monday – Friday 8 a.m. - 4:30 p.m.
Feedstock used: yard trimmings, stable manures, redwood grindings
Delivery available. Self – haul price $20/yard, bags available for small quantities.

Before and After --- Master Bathroom Remodeling

2011-09-04T06:04:00-07:00

The old master bathroom is dark and ugly, with only one sink and w/o tub.

We wanted a master bathroom that is modern and bright, and at the same time with a sense of Zen. We also wanted it to have two sink and separate shower and tub. Three months and a lot of work later, we have what we wanted and it totally worthies it since we are now enjoying it everyday.

I love everything about the new bathroom. But the top three on my lists are:

1. Shower panel with rain head --- it makes each shower much enjoyable that I really miss it every time I go travel even when I stay in 5-star hotels.

2. The recessed medical cabinets from Pottery Barn --- they are a little pricey but well-worth it, because they look nice and at the same time can hide all my stuff. They are the reason why the counter is always clutter-free.

3. Maax Free Standing Air Tub --- It makes every bath a really relaxing experience and super easy to clean and maintain (no harder than a usual soaking tub, no worry for mold).

Gift Buying Guide for Your Family and Friends in China

2011-08-05T06:30:00-07:00

China is a fast-changing country. Not surprisingly, the list of popular gifts for Chinese family, relatives and friends keeps on changing as well. When I return to China, I always do some extensive research to gather an updated shopping list of gifts to buy. Here is a summary based on various gift-giving guides and online forum postings for my recent trip to China.

1. Parents/Elderly Relatives

* Vitamin Supplements

* Wisconsin Ginseng

Health-related gifts have been always appreciated in China. Even though the same product may be available in China, items bought oversea are considered as of higher quality.

2. Women

* Skin Care/Cosmetics

Most Western brands sold in department store such as Clinique, Estee Lauder and Lancome are very much adored by Chinese women. However, if you happen to get any free gift with purchase, don't gift that alone. Many people know that you get it for free, so it would be very insulting if you just give the free gift as a gift.

* Jewelry

If you really want to make a strong impression, buy from well-known brands such as Tiffany and Cartier. However, it is not a good idea to buy non-name brand pearl, which is typically much cheaper in China, or any gold jewelry that is 14K or lower, as gold jewelry in China is typically 18K or higher.

* Chocolate/Candies

The following items are especially popular: Lindt chocolate, Ferrero Rochet Fine Hazelnut chocolate, Almond Roca.

* Handbag/Sun Glass/Clothing

Well-known name brands including Adidas, LV, Gucci, Chanel, Ralph Lauren, COACH, CK, Guess, AE, Gap, Levis, Express, Bebe, Nine West, Juicy Couture. Believe it or not, although most products are made in China, it is usually much cheaper to buy the name brands in US.

3. Men

* Cigarette / Zippo lighter

A good portion of men in China smoke. Brand name cigarette and Zippo lighter can be good gift for them.

* Red Wine

A good choice is any Napa Valley red wine.

* State Quarters / Yearly Stamp Collection

* Name brand watch/belt/shoe

4. Children

* Chocolate/Candies

* Name-brand Clothing

Most well-known name brands are good choices.

* Disney-brand toys / Educational toys

Display LaTeX Math on Drupal with MathJax

2011-07-18T20:19:00-07:00

MathJax seems to be the emerging standard for displaying math on the Web at this moment. It is supported by American Mathematical Society and American Physical Society, and has already been adopted by major math related discussion venues such as Physics Forums and Stack Exchange. MathJax displays math using CSS and Web fonts instead of images, so the quality is very high and is resize-friendly. Below are some examples:

Inline math: the geometric product $\boldsymbol{uv}$ of vectors $\boldsymbol{u}$ and $\boldsymbol{v}$ is $\boldsymbol{u}\cdot\boldsymbol{v} + \boldsymbol{u}\wedge\boldsymbol{v}$, where $\boldsymbol{u}\cdot\boldsymbol{v}$ is the inner product and $\boldsymbol{u}\wedge\boldsymbol{v}$ is the outer product.

Display math: the rotation of vector $\boldsymbol{u}$ by angle $\theta$ in plane $\boldsymbol{i}$ is $$R_{\boldsymbol{i}\theta}(\boldsymbol{u}) = e^{-\boldsymbol{i}\theta/2}\boldsymbol{u}e^{\boldsymbol{i}\theta/2}$$

Some random expressions copied directly from the short math guide for latex: $$\begin{pmatrix} \alpha& \beta^{}\ \gamma^{}& \delta \end{pmatrix}$$ $$\frac{{\displaystyle\sum_{n > 0} z^n}} {{\displaystyle\prod_{1\leq k\leq n} (1-q^k)}}$$ $$2^k-\binom{k}{1}2^{k-1}+\binom{k}{2}2^{k-2}$$

Finally, an obligatory integral: $$\int !!! \int_D f(x,y)\,dx\,dy$$

The math input could be either LaTeX or MathML, embedded in regular HTML text. MathJax is a Javascript library, so it works at the browser's side. After the HTML is rendered, the MathJax code scans the output, find pieces of text marked by user defined math delimiters (more on that later), and replaces them by typesetted math. The math may take a couple of seconds to show up depending on the browser and the network speed. The MathJax library can be fetched from content delivery network (CDN), so it is fairly simple to add MathJax to any Web site. Here is how I did it for this Drupal 6 site.

There is a Drupal module for MathJax that loads MathJax from CDN for every Drupal page. The module also allows Drupal pages to be selectively MathJax enabled according to some URL patterns. By default, all pages are MathJax enabled except for these paths: admin/, node/add/, node//edit/, which is reasonable. However, path based selective enabling is still a bit crude, because most pages would not contain math, so MathJax simply slows down page rendering for most pages without any benefit. It would be great if the selection can also be tag based, so only the posts tagged as "math" and maybe the front page need to incur such slow down. Another useful feature would be to allow MathJax configuration within the module. The default configuration of MathJax is not very reasonable for Drupal, so users have to add the configurations manually to the page template at this time. Updated Aug.25, 2011:* Julou, the author of the module, has responded to my feature request and added a text area for MathJax configuration in the development version of the module. I have tested it and it works.

One of the most annoying default settings of MathJax is the definition of math delimiters. For inline math, the default delimiters are ( ), and for math on its own line, the default delimiters are [ ]. Because ( ) and [ ] are so commonly used in regular non-math text, this default setting is unreasonable for Drupal, as MathJax will remove the parentheses and change the text font to be math like for a Drupal page. Double dollar sign is okay, but to show the regular dollar sign properly, it needs to be escaped by reverse backslash, which is not the default setting of MathJax. Also, the default message display of MathJax is a bit too much for my taste, and I minimized them a bit.

If your site has the print module installed, you need to copy the file print.tpl.php to your theme templates directory, and add the same as above so the print friendly version also show math properly.

For more MathJax configuration options, please refer to the MathJax documentation

Upgrade Drupal with Almost Zero Down Time

2011-07-16T08:17:00-07:00

This site was setup in 2007 with Drupal 5, and it has not been upgraded until today. I did not feel the need to upgrade since this is just a personal blog. A few days ago I wanted to install a module, but Drupal version 6 is required, so I thought it's finally the time to upgrade Drupal.

This Drupal upgrade tutorial suggests to first take down the site to upgrade, but I do not like that. It turns out that more than a few days of work are required to bring back a fully functional upgraded site, so taking the site down to upgrade is not a good idea. Below is what I did to upgrade this site from version 5.3 to 6.22 with almost zero down time (well, maybe a few seconds, I didn't time it).

The idea is simple: keep the old site running, setup a test site with a copy of the data, upgrade at test site, fix any problems there, delete the old site when done, and move the test site to be the production site.

First, the old version 5 site needs to be prepared. This means upgrading it from whatever old version to the latest version 5.22. Since this preparation is a minor upgrade, there is no need to bring down the site. Just download the 5.22 tar ball and unpack it over the existing Drupal directory. Run yoursite.com/update.php to update the database. Then download and install update_status module. It will tell you which third-party modules need to be upgraded. Download, unpack, run update.php, repeat, until all modules are at the lastest 5.x versions. Finally, uninstall update_status module, since it is included in Drupal 6 core.

To setup a test site, first create a test.yoursite.com sub-domain using your hosting company's tool. Most Web hosting companies allow sub-domains free of charge, and often provide easy to use tools for managing them (if yours does not, switch host company!). Download and unpack the latest Drupal 6 tar ball to a directory in the Web directory, and point the test.yoursite.com sub-domain to the directory using the hosting company's tool. Point the browser to test.yoursite.come, the Drupal installation page should show up. Now stop, DO NOT go though the installation process, for we are upgrading, not installing. Because this upgrade method requires two Drupal sites (e.g. one version 5 and one 6) running at the same time, we need to have two databases. Your hosting company should allow the creation of new databases free of charge.

First, we need to check the minimum system requirement of the new version of Drupal, especially the database requirement. This can be found at yoursite.com/admin/reports/status page. Here, we are reaping the benefit of keeping the old site running while upgrading the test site: we can look up system information easily. In my case, the old site is using MySQL version 4.0, which is older than the minimum requirement of Drupal 6. Fortunately, newly created databases using my hosting company's tool are MySQL version 5.0. One thing needs to be careful about is the character encoding of the new database. Make sure it is using utf-8 for charset, and the collation is utf8_general_ci. The later is often not the default, so we need to set it under Operations/Collations in phpMyAdmin, a MySQL management tool most hosting companies provide.

Now we need to roll over the data. This can be done simply with phpMyAdmin. First, login as the primary administrator (the first user, uid=1) at the old Drupal site, and then dump its database using phpMyAdmin. Basically, we need to export all the tables. If the tables are big, you may want to export a few tables at a time. For some tables, such as cache, cache_*, sessions, search_*, and watchdog, you may want to export the structure only, instead of both structure and data, as these data can be dynamically regenerated.

The exported database consists of a bunch of compressed SQL text files. Using phpMyAdmin, browse to the newly created empty database, run these SQL files to import all the tables. After the new database is ready, we need to edit the new Drupal directory's sites/default/settings.php file. This file is not there initially. There are two ways to create it: a). Copy the file from old Drupal directory if there are a lot of customizations done there; b). Rename the file default.settings.php to settings.php. I took the later approach. Now in settings.php, find the line looks like \$db_url = 'mysql://username:password@hostname/databasename';, and replace with the correct information for the new database. Save the file, now the new Drupal knows where the database is. Next step is to copy over the old Drupal directory's files directory, and redo any customizations done for .htaccess and robot.txt. Point the browser to test.yoursite.com/update.php, and follow instructions to update the database. There might be some warnings, but it should be fine if there's no error.

After Drupal core is upgraded. Download the third-party modules, and install them. After installing each one, don't forget to run update.php again. We again look at the running old site to see what modules are needed. Otherwise, one has to write down a long list of module names first. I still cannot believe how many modules are necessary to run a simple site like this. For fighting spams alone, I installed seven modules. Maybe it is an overkill, but it does not hurt to be on the prudent side.

Upgrading modules are easy and relatively quick. The big time sink is upgrading the theme. This site uses a sub-theme of zen. It turns out that the theming systems of Drupal 5 and 6 are quite different, so we cannot just copy the old theme over and expect it to work. All the customizations have to be redone, which means hacking the template files and the CSS files all over again. Having a working old site running is again a big help. Because the tag ids and classes are all different, it is necessary to use Firebug to track and match the styles of the two versions while hacking the CSS files.

After the new site is tested to be working properly, it is the time to bring the old site off-line and bring the new site online. All it requires is to issue a rm -rf command to delete the old Drupal directory, followed by a mv command to move the new Drupal directory to the old one, so that amounts to a couple seconds of down time. Not bad.

3 Time-Saving Tools for Busy Professionals

2011-07-16T08:06:00-07:00

As a busy professional, I am consistently looking for ways to save time. Below are 3 time-saving tools that I have found most useful.

1. Chore Buster

With two busy professionals in the same household, dividing chores around the house can be tricky. Dividing chores in a fixed way is one solution. But one can quickly get bored of doing the same thing everyday.

With this neat free online tool, we get automatic daily chore assignment based on a pre-defined list of house chores and preference of chores for each person. It is really fun to check off the items at the end of the day. Besides ensure fairness of the chore assignment, the tool also serves as a reminder for chores that only happens once in a while.

2. Mint

As the CFO of the family, I do need to keep track of our financial situation. However, I don't really have much time to do so. Mint has been a great help in this regards.

3. Farm Fresh to You

Both Huahai and I enjoy cooking. However, weekly grocery shopping can be time-consuming. This service basically gets organic produce delivery directly to my home from a local organic farmers. It costs $31.5 a box for a week of organic vegetable and fruit.

We now only do grocery shopping every other weeks, saving both time and fuel. One thing that I like about the service is that I can exclude items so that I won't get anything that I don't like.

If you decide to try this service out, don't forget to mention promotion code 2798 and our customer number 236470 to get 20% off your first box.

How to Share Mouse and Keyboard across Machines

2011-07-11T00:45:00-07:00

If you have multiple machines but only one set of mouse and keyboard, then you must check out this neat little tool called Synergy. It allows you to seamlessly share mouse and keyboard across machines. It even allows you to copy and paste text across machines.

I have been using it for a few years now and have recommended it to many already. We all absolutely love it. My current set up is a desktop with two monitors and a set of keyboard and mouse connecting to it. Then I have a laptop with another monitor sharing the keyboard and mouse via Synergy with my desktop. This way I can use two machines and three monitors at the same time without any interruption.

I simply cannot imagine my life w/o Synergy. It is technically straightforward but really useful!

Fix Drupal Search

2011-07-10T19:18:00-07:00

Today I tried to search "clojure" using the search box at the top right corner, but could not find anything. I know I wrote a few posts on Clojure recently, so there must be something wrong with the search functionality here. This is a Drupal site, and this should be an easy fix. Indeed, I quickly figured out the problem and solved it.Here is what Idid.

After I login to the administrative interface, I saw lots of cron related errors at the /admin/logs/watchdog page: "Cron has been running for more than an hour and is most likely stuck." A google search suggests that this is a common problem. One of the possible reasons is that there are too many things for cron to do so that it cannot finish them within the time limit or it runs out of the memory. The solution is to reduce the amount of work a cron job has to do. Since the indexing of the content on this site is done with a cron job, I checked the indexing setting at /admin/settings/search page. Sure enough, the option Items to index per cron run: was set at 100, so I changed it to 10 instead.

I then tried to manually run cron job at /admin/logs/status page, but it didn't work. It turned out that it is necessary to clear the cron related variables in the MySQL database Drupal uses. To do that, I logined into my hosting company's phpmyadmin interface, searched for "cron_last" and "cron_semaphore". There was no "cron_semaphore", but there was a "cron_last" entry, so I deleted it. Manually ran cron again, now it said "Cron ran successfully". After several manual runs to get the indexing status to be 100% done, now I can do a search on "clojure", and get the posts to show up. Nice.

Review on Hex Bug as Cat Toy

2011-07-10T06:04:00-07:00

I am consistantly looking for new ways to entertain the cats, as they get bored with their toys very quickly. One new toy I have recently bought for them is Hex Ant.

I tried it during the cats outdoor time. Mayori completely ingored it, while Tom was obviously attracted to it immediately.

He tried to smell it.

Unfortunately, his interest towards the new toy didn't last long. He quickly walked away as soon as he found out it is not a real bug.

I thus don't recommend it for cat toy. However, it is an interesting piece of toy for myself. So I still recommend it as a human toy.

Review of The Refined Feline Lotus Cat Tower

2011-07-04T09:06:00-07:00

The old cat tree of Mayori and Tom were bought nearly 4 years ago and has started to show its age. I have been shopping for a new cat tree for a while now. I need something that is nicer than the typical carpet covered cat tree and goes well with the rest of the house. At the same time, I want to get something that Mayori and Tom can really enjoy. Unfortunately, there is very limited number of such choices on the market. One of them is Refined Feline Lotus Cat Tower

According to the online reviews, most of the cats and the cat owners like this product. So we brought one for the cats to try out.

Tom was very curious to check the package out when it arrived. He pointed out some minor damage on the package itself.

But he liked new cat tree already once we unpackaged it and immediately claimed one piece as his own.

Mayori seemed to be satisfied too and kept on inspecting the installation process.

And she claimed the top as soon as the installation was completed. (The installation took less than 2 hours and was not difficult according to Huahai.)

Tom claimed the cube at the bottom.

Comparing to the old cat tree, the new Refine Feline Lotus Cat Tower is obviously much nicer and looks more like a human furniture. It goes really well with the rest of our bedroom furniture. It is definitely worthy considering for one who is looking to find a cat furniture with great style.

Cat with Evil Eyes

2011-07-04T07:58:00-07:00

Mayori is mostly a very sweet girl. However, she can have evil eyes as well. Look at the follow photo when she wants to hunt to the toy mouse. She certainly looks more like a leopard than a domestic cat!

Cat Boxing

2011-07-04T07:44:00-07:00

Mayori and Tom grow up together ever since they were still little kittens in the shelter. However, they still love to fight, especially for the best spot for window-watching the outside.

Samba Cat Dancer

2011-07-04T07:35:00-07:00

Tom shows off his Samba dance skill.

Cat with Impeccable Taste

2011-07-04T07:17:00-07:00

As a cat, Mayori has impeccable taste, as represented by her favorite furniture in the house --- a Bertoia Diamond Lounge chair.

It is certainly a modern classic and a piece of highly functional cat furniture as well – Mayori can attack Tom easily from all different angels and only need to defend her from one angel. What a smart cat!

Cat with Big Paws

2011-07-04T06:56:00-07:00

Tom shows off his catch and his big paws.

Cat Drinking from Wine Glass

2011-07-04T06:44:00-07:00

Mayori loves drinking fresh-served water, especially from a wine glass. Obviously she believes that is the right way to drink water for a cat princess.

Cat in a Drawer

2011-07-04T06:36:00-07:00

Mayori loves to play hide-and-seek with us. The top drawer of our dresser is now her new hiding place.

Cat Pizza Delivery

2011-07-04T06:31:00-07:00

Cat in a pizza box?

Either Mayori believes that she tastes better than the fresh baked version or that she believe that she needs a new bed smelled like fresh baked pizza. Either way she obviously enjoys the pizza box.

How to redeem British Airways mileages for partner airline flights

2011-06-15T05:35:00-07:00

I have accumulated over 60,000 miles on British Airlines Executive Club reward account for a while now, but keep on having problem redeeming them. First of all, it is difficult to find any reward flight available; secondly, the fee is extremely high --- for a flight from SFO to LHR, I would have to pay 50000 miles + over $600 fee, while the same flight costs less than $1000 when paying with cash.

Over the weekend, I have investigated the problem and found out that one of the best ways to spend my BA miles is to redeem for British Airways' partner airline flights. However, it is not straightforward to do so, as BA would show its own flights with steep fee first, except when American Airlines and a couple of other partner airlines operate the same route.

The easiest way I found is the following:

1. First, check AA's route map to find out flight routes operated by AA:

2. Once you find a route of interest, go to BA's website

3. Click on the tab "Executive Club", choose "Spending BA miles" on the drop-down menu, then click on "Book with BA miles" on the left-hand side menu.

4. Log in and search with the desired route and time.

For instance, I am interested in booking a flight between SFO and HNL, because I know there are only flights operated by AA for the route.

You will see a screen similar to the following:

5. Now you can either click on "Search" or "Include Parnter" for flights operated by AA.

If you see the red error message "Unfortunately our systems are not responding, so we are unable to process your request at the moment. We apologies for this inconvenience and suggest you try again later." Don't panic. Just keep on clicking and it will go through eventually.

With the above method, I was able to book two reward tickets from SFO to HNL with 35,000 miles + $10 fee each.

My solutions for problems No. 76-100 on 4clojure.com

2011-06-05T05:36:00-07:00

Finally done with all the 100 problems listed on 4clojure.com so far :-). When new problems appears there, I will probably do them when I have some time to kill, but I will not post my solutions here any more. If I found interesting programming exercises, I may submit to 4clojure as well.

This has been a great learning experience. I become very familiar with the core Clojure functions, especially the great sequence library. The functional way of writing code is so much fun, I wish I have been exposed to it earlier. Now I am hooked and would like to learn more. It's a pity there ain't many functional algorithm books around. The one I found is Chris Okasaki's "Purely Functional Data Structures", but the exposition language is Standard ML. Let's hope it can be easily translated into Clojure...

; 77: Write a function which finds all the anagrams in a vector of words. A
; word x is an anagram of word y if all the letters in x can be rearranged in
; a different order to form y. Your function should return a set of sets, where
; each sub-set is a group of words which are anagrams of each other. Each
; sub-set should have at least two words. Words without any anagrams should not
; be included in the result.
; (= (__ ["meat" "mat" "team" "mate" "eat"]) #{#{"meat" "team" "mate"}})
; (= (__ ["veer" "lake" "item" "kale" "mite" "ever"])
;   #{#{"veer" "ever"} #{"lake" "kale"} #{"mite" "item"}})
(fn [coll]
  (->> (group-by frequencies coll)
    (vals)
    (filter #(> (count %) 1))
    (map set )
    set))
; anagrams have the same distribution of characters

; 79: Write a function which calculates the sum of the minimal path through a
; triangle. The triangle is represented as a vector of vectors. The path should
; start at the top of the triangle and move to an adjacent number on the next
; row until the bottom of the triangle is reached.
; (= (__ [  [1]
          ;[2 4]
         ;[5 1 4]
        ;[2 3 4 5]])
   ;(+ 1 2 1 3)
   ;7)
(fn [triangle]
  (apply min ((fn path-sum [p]
                (concat
                  (if (= (count triangle) (count p))
                    [(reduce + (map-indexed #(get-in triangle [%1 %2]) p))]
                    (let [x (last p)]
                      (concat
                        (path-sum (conj p x))
                        (path-sum (conj p (inc x))))))))
              [0])))
; We enumerate all possible paths. The next step in a path can only go to the
; same or the plus one row index as the previous step, so the paths form a
; binary tree. We walk the tree recursively, building a row index vector p for
; each path.

; 81: Reimplement set intersection
#(set (filter %1 %2))
; sets are functions too, so this works

; 82: A word chain consists of a set of words ordered so that each word differs
; by only one letter from the words directly before and after it. The one
; letter difference can be either an insertion, a deletion, or a substitution.
; Here is an example word chain:
; cat -> cot -> coat -> oat -> hat -> hot -> hog -> dog
; Write a function which takes a sequence of words, and returns true if they
; can be arranged into one continous word chain, and false if they cannot.
; (= false (__ #{"cot" "hot" "bat" "fat"}))
; (= true (__ #{"spout" "do" "pot" "pout" "spot" "dot"}))
(fn [word-set]
  (letfn [(edit-dist [a b]
            (cond
              (not (or a b)) 0
              (not b) (count a)
              (not a) (count b)
              :else (let [ra (next a) rb (next b)]
                      (if (= (first a) (first b))
                        (edit-dist ra rb)
                        (+ 1 (min
                               (edit-dist ra rb)
                               (edit-dist ra b)
                               (edit-dist a rb)))))))
          (find-paths [graph start seen]
            (if (seen start)
              seen
              (for [n (graph start)]
                (find-paths graph n (conj seen start)))))]
    (let [graph (into {}
                      (for [s word-set]
                        [s (filter #(= 1 (edit-dist s %)) word-set)]))]
      (if (some (fn [w]
                  (some #(= word-set %)
                        (flatten (find-paths graph w #{}))))
                word-set)
        true false))))
; This problem consists of two sub-problems: A. Determine the edit distance
; between two strings. For brevity, we just used the standard recursive
; algorithm instead of dynamic programming. B. For the graph of strings
; connected by edges of edit distance 1, find a simple (no loop) path that
; goes through all strings once and only once. The graph is represented as
; a map of adjacent node lists. We enumerate all simple paths in the graph
; until we found one going through all nodes.

; 84: Write a function which generates the transitive closure of a binary
; relation. The relation will be represented as a set of 2 item vectors.
; (let [divides #{[8 4] [9 3] [4 2] [27 9]}]
;   (= (__ divides) #{[4 2] [8 4] [8 2] [9 3] [27 9] [27 3]}))
; (let [progeny
;       #{["father" "son"] ["uncle" "cousin"] ["son" "grandson"]}]
;         (= (__ progeny)
;              #{["father" "son"] ["father" "grandson"]
;                     ["uncle" "cousin"] ["son" "grandson"]}))
(fn [relation]
  (letfn [(expand [r]
            (let [m (into {} r)]
              (->> (concat
                     r
                     (for [[k v] m]
                       (when-let [nv (m v)] [k nv])))
                (filter identity)
                set)))
          (first-consecutive [pred [f & rs]]
            (when rs
              (if (pred f (first rs))
                f
                (recur pred rs))))]
    (first-consecutive = (iterate expand relation))))
; we iteratively expand the set of transitive relation, until the set no
; longer changes

; 85: Write a function which generates the power set of a given set. The power
; set of a set x is the set of all subsets of x, including the empty set and x
; itself.
; (= (__ #{1 :a}) #{#{1 :a} #{:a} #{} #{1}})
(fn [s]
  (reduce
    (fn [init e]
      (set (concat init (map #(conj % e) init) [#{e}])))
    #{#{}} s))
; we just add one element at a time

; 86: Happy numbers are positive integers that follow a particular formula: take
; each individual digit, square it, and then sum the squares to get a new number
; Repeat with the new number and eventually, you might get to a number whose
; squared sum is 1. This is a happy number. An unhappy number (or sad number) is
; one that loops endlessly. Write a function that determines if a number is
; happy or not.
; (= (__ 7) true)
; (= (__ 986543210) true)
(fn [x]
  (letfn [(digits [n]
            (for [y (iterate (partial * 10) 1) :while (<= y n)]
              (rem (int (/ n y)) 10)))
          (sqr-sum [ds]
            (reduce + (map #(* % %) ds)))]
    (let [r (some #{1 4} (iterate (comp sqr-sum digits) x))]
      (cond
        (= 1 r) true
        (= 4 r) false))))
; it turns out that 4 is a sad number, as it results into an infinite loop

; 88: Write a function which returns the symmetric difference of two sets. The
; symmetric difference is the set of items belonging to one but not both of
; the two sets.
; (= (__ #{1 2 3 4 5 6} #{1 3 5 7}) #{2 4 6 7})
#(set (remove (set (filter %1 %2)) (into %1 %2)))
; we remove the intersection from the union

; 89: Starting with a graph you must write a function that returns true if it
; is possible to make a tour of the graph in which every edge is visited exactly
; once.  The graph is represented by a vector of tuples, where each tuple
; represents a single edge.  The rules are:
; - You can start at any node.
; - You must visit each edge exactly once.
; - All edges are undirected.
; (= true (__ [[1 2] [2 3] [3 4] [4 1]]))
; (= false (__ [[1 2] [2 3] [2 4] [2 5]]))
; (= false (__ [[:a :b] [:a :b] [:a :c] [:c :a] [:a :d] [:b :d] [:c :d]]))
; (= true (__ [[:a :b] [:a :c] [:c :b] [:a :e] [:b :e] [:a :d] [:b :d]
;              [:c :e] [:d :e] [:c :f] [:d :f]]))
(fn [edge-list]
  (let [graph (apply merge-with
                #(into %1 %2)
                (apply concat
                  (map-indexed
                    (fn [i [k v]]
                      [{k #{{:node v :index i}}}
                       {v #{{:node k :index i}}}])
                    edge-list)))]
    (if (some
          (fn [node]
            (some
              identity
              (flatten
                ((fn visit [n vs]
                   (if (every? #(vs (:index %)) (graph n))
                     (if (every? identity vs) true false)
                     (for [x (graph n)]
                       (when-not (vs (:index x))
                         (visit (:node x) (assoc vs (:index x) true))))))
                 node (vec (repeat (count edge-list) false))))))
          (set (apply concat edge-list)))
      true false)))
; This problem looks similar to problem 82 as both are graph traversals, but
; the graphs are quite different. Here redundent edges exist, so we cannot use
; a set or a map to track edge visits, we instead use a vector of booleans.
; Also, the condition is to traverse all edges instead of all nodes. We again
; use a map of adjacent node lists as the graph, but supplement each adjacent
; node with the index of the corresponding edge. Finally, here a node can be
; visited multiple times, and we terminates a path at a node only when all of
; its edges have already been visited.

; 91: Given a graph, determine whether the graph is connected. A connected
; graph is such that a path exists between any two given nodes.
; -Your function must return true if the graph is connected and false otherwise.
; -You will be given a set of tuples representing the edges of a graph. Each
;  member of a tuple being a vertex/node in the graph.
; -Each edge is undirected (can be traversed either direction).
;  (= false (__ #{[1 2] [2 3] [3 1] [4 5] [5 6] [6 4]}))
;  (= true (__ #{[1 2] [2 3] [3 1][4 5] [5 6] [6 4] [3 4]}))
(fn [edge-list]
  (let [graph (apply merge-with
                #(into %1 %2)
                (apply concat
                  (map (fn [[k v]] [{k #{v}} {v #{k}}]) edeg-list)))]
    (if (some #(= (count %) (count graph))
              (flatten
                ((fn paths [node seen]
                   (if (seen node)
                     seen
                     (for [x (graph node)]
                       (paths x (conj seen node)))))
                 (ffirst graph) #{})))
      true false)))
;  This graph traversal problem is simpler than both 82 and 89. We only  need
;  to start searching from any one of the nodes instead of all nodes. But the
;  pattern of the code is similar.

; 92: Write a function to parse a Roman-numeral string and return the number it
; represents. You can assume that the input will be well-formed, in upper-case,
; and follow the subtractive principle. You don't need to handle any numbers
; greater than MMMCMXCIX (3999), the largest number representable with
; ordinary letters.
; (= 827 (__ "DCCCXXVII"))
; (= 48 (__ "XLVIII"))
(fn [s]
  (let [snum {[\C \M] 900  [\C \D] 400 [\X \C] 90
             [\X \L] 40 [\I \X] 9 [\I \V] 4}
        nums {\I 1 \V 5 \X 10 \L 50 \C 100 \D 500 \M 1000}]
    (letfn [(sum-snum [[f & r]]
                      (if f
                        (+ (if-let [n (snum [f (first r)])]
                             n 0)
                           (sum-snum r))
                        0))
            (del-snum [[f & r]]
                         (when f
                           (if (snum [f (first r)])
                             (del-snum (rest r))
                             (cons f (del-snum r)))))]
      (reduce + (sum-snum s) (map nums (del-snum s))))))
; We first find and sum the special numbers (4, 9, etc), remove them and sum
; the rest.

; 93: Write a function which flattens any nested combination of sequential
; things (lists, vectors, etc.), but maintains the lowest level sequential
; items. The result should be a sequence of sequences with only one level of
; nesting.
; (= (__ '((1 2)((3 4)((((5 6))))))) '((1 2)(3 4)(5 6)))
(fn pf [coll]
  (let [l (first coll) r (next coll)]
    (concat
      (if (and (sequential? l) (not (sequential? (first l))))
        [l]
        (pf l))
      (when (sequential? r)
        (pf r)))))
; this is just a slight modification of the solution to problem 28.

; 94: The game of life is a cellular automaton devised by mathematician John
; Conway.  The 'board' consists of both live (#) and dead ( ) cells. Each cell
; interacts with its eight neighbours (horizontal, vertical, diagonal), and its
; next state is dependent on the following rules: 1) Any live cell with fewer
; than two live neighbours dies, as if caused by under-population.  2) Any live
; cell with two or three live neighbours lives on to the next generation.  3)
; Any live cell with more than three live neighbours dies, as if by overcrowding
; . 4) Any dead cell with exactly three live neighbours becomes a live cell, as
; if by reproduction.  Write a function that accepts a board, and returns a
; board representing the next generation of cells.
;(= (__ ["      "
        ;" ##   "
        ;" ##   "
        ;"   ## "
        ;"   ## "
        ;"      "])
   ;["      "
    ;" ##   "
    ;" #    "
    ;"    # "
    ;"   ## "
    ;"      "])
(fn [board]
  (let [offsets [[-1 -1] [-1 0] [-1 1]
                 [0 -1] [0 1]
                 [1 -1] [1 0] [1 1]]
        height (count board)
        width (count (first board))
        get-state (fn [[x y] [dx dy]]
                    (let [c (+ x dx) r (+ y dy)]
                      (if (or (< c 0) (= c width) (< r 0) (= r height))
                        \space
                        (get-in board [r c]))))
        count-lives (fn [p]
                      (reduce + (map #(if (= \# (get-state p %)) 1 0) offsets)))
        next-state (fn [s p]
                     (let [n (count-lives p)]
                       (if (or (= n 3)
                               (and (= s \#) (= n 2)))
                         \#
                         \space)))]
    (->> (for [y (range height) x (range width)]
           (next-state (get-in board [y x]) [x y]))
      (partition width)
      (map #(apply str %))
      vec)))
; This is straight-forward. The only tricky part is to remember that the order
; of paramaters for the get-in function and the x-y coordinates is opposite to
; each other.

; 95: Write a predicate which checks whether or not a given sequence represents
; a binary tree. Each node in the tree must have a value, a left child, and a
; right child.
; (= (__ '(:a (:b nil nil) nil)) true)
; (= (__ '(:a (:b nil nil))) false)
; (= (__ [1 nil [2 [3 nil nil] [4 nil nil]]]) true)
(fn bt? [t]
  (if (or (not (sequential? t))
          (and (= 3 (count t))
               (bt? (second t))
               (bt? (last t))))
    true false))
; I think one of the unit tests of the problem is wrong:
; (= (__ [1 [2 [3 [4 false nil] nil] nil] nil]) false)
; why shouldn't "false" be a legal tree node, or why should leaf have to be nil?

; 96: Let us define a binary tree as "symmetric" if the left half of the tree
; is the mirror image of the right half of the tree. Write a predicate to
; determine whether or not a given binary tree is symmetric.
; (= (__ '(:a (:b nil nil) (:b nil nil))) true)
; (= (__ '(:a (:b nil nil) nil)) false)
; (= (__ [1 [2 nil [3 [4 [5 nil nil] [6 nil nil]] nil]]
          ;[2 [3 nil [4 [6 nil nil] [5 nil nil]]] nil]]) true)
(fn [t]
  ((fn mir? [l r]
     (if (or (= nil l r)
             (and (= (first l) (first r))
                  (mir? (second l) (last r))
                  (mir? (last l) (second r))))
       true false))
   (second t) (last t)))

; 97: Pascal's triangle is a triangle of numbers computed using the following
; rules:
; - The first row is 1.
; - Each successive row is computed by adding together adjacent numbers in the
;   row above, and adding a 1 to the beginning and end of the row.
; Write a function which returns the nth row of Pascal's Triangle.
; (= (map __ (range 1 6))
   ;[     [1]
        ;[1 1]
       ;[1 2 1]
      ;[1 3 3 1]
     ;[1 4 6 4 1]])
(fn [n]
  (nth (iterate
         (fn [pre]
           (vec
             (concat
               [1]
               (map (fn [[f s]] (+ f s)) (partition 2 1 pre))
               [1])))
         [1])
       (dec n)))

My solutions for problems No. 51-75 on 4clojure.com

2011-05-25T01:59:00-07:00

This post continues the previous one, on my solutions for small clojure programming problems on 4clojure.com. Doing these problems seems to be addictive as I could not seem to stop myself. The site recently added a golf league feature, so one can see how short one's own solution compared with others. If a lot of people got a much shorter solution than yours, you know you are not thinking in the right way. This little competition makes the site even more attractive. Anyhow, the code is here:

; 53: Given a vector of integers, find the longest consecutive sub-sequence of
; increasing numbers.  If two sub-sequences have the same length, use the one
; that occurs first. An increasing sub-sequence must have a length of 2 or
; greater to qualify.
; (= (__ [1 0 1 2 3 0 4 5]) [0 1 2 3])
; (= (__ [7 6 5 4]) [])
(fn [coll]
  (->> (partition 2 1 coll)
    (partition-by #(- (second %) (first %)))
    (filter #(= 1 (- (second (first %)) (ffirst %))))
    (reduce #(if (< (count %1) (count %2)) %2 %1) [])
    flatten
    distinct))
; we first create a list of neighoring pairs, partition them by their pair
; differences, keep those with difference 1, finally return the longest one

; 54: Write a function which returns a sequence of lists of x items each.
; Lists of less than x items should not be returned.
; (= (__ 3 (range 8)) '((0 1 2) (3 4 5)))
; forbidden: partition, partition-all
(fn partition2 [n coll]
(when (<= n (count coll))
(cons (take n coll) (partition2 n (drop n coll)))))
; we recursively take n items till not enough items

; 55: Write a function that returns a map containing the number of occurences
; of each distinct item in a sequence.
; (= (__ [1 1 2 3 2 1 1]) {1 4, 2 2, 3 1})
; forbidden: frequencies
(fn [coll]
(let [gp (group-by identity coll)]
(zipmap (keys gp) (map #(count (second %)) gp))))
; note a map entry is just a two item vector, first item is the key, the
; second item is the value

; 56: Write a function which removes the duplicates from a sequence. Order of
; the items must be maintained.
; (= (__ [1 2 1 3 1 2 4]) [1 2 3 4])
; forbidden: distinct
(fn [coll]
  ((fn step [[x & xs] seen]
     (when x
       (if (seen x)
         (step xs seen)
         (cons x (step xs (conj seen x))))))
   coll #{}))
; we recursively go through the sequence, use a set to keep track of items
; we've seen, only return those we have not seen before.

; 58: Write a function which allows you to create function compositions. The
; parameter list should take a variable number of functions, and create a
; function applies them from right-to-left.
; (= [3 2 1] ((__ rest reverse) [1 2 3 4]))
; forbidden: comp
(fn [x & xs]
  (fn [& args]
    ((fn step [[f & fs] a]
       (if fs
         (f (step fs a))
         (apply f a)))
     (cons x xs) args)))
; step function takes the function list and the arguments, recursively builds
; an ever deeper call stack till at the end of the list, where the right most
; function is called with the given arguments.

; 59: Take a set of functions and return a new function that takes a variable
; number of arguments and returns sequence containing the result of applying
; each function left-to-right to the argument list.
; (= [21 6 1] ((__ + max min) 2 3 5 1 6 4))
; forbidden: juxt
(fn [x & xs]
(fn [& args]
(map #(apply % args) (cons x xs))))

; 60: Write a function which behaves like reduce, but returns each
; intermediate value of the reduction. Your function must accept either two
; or three arguments, and the return sequence must be lazy.
; (= (take 5 (__ + (range))) [0 1 3 6 10])
; (= (__ conj [1] [2 3 4]) [[1] [1 2] [1 2 3] [1 2 3 4]])
; forbidden: reductions
(fn reductions2
  ([f init [x & xs]]
   (cons init (lazy-seq (when x (reductions2 f (f init x) xs)))))
  ([f coll]
   (reductions2 f (first coll) (rest coll))))

; 61: Write a function which takes a vector of keys and a vector of values
; and constructs a map from them.
; (= (__ [:a :b :c] [1 2 3]) {:a 1, :b 2, :c 3})
; forbidden: zipmap
#(into {} (map vector %1 %2))

; 62. Given a side-effect free function f and an initial value x
; write a function which returns an infinite lazy sequence of x,
; (f x), (f (f x)), (f (f (f x))), etc.
; (= (take 5 (__ #(* 2 %) 1)) [1 2 4 8 16])
; forbidden: iterate
(fn iterate2 [f x]
(cons x (lazy-seq (iterate2 f (f x)))))
; it turns out that clojure's own implmentation is the same

; 63. Given a function f and a sequence s, write a function which returns a
; map. The keys should be the values of f applied to each item in s. The value
; at each key should be a vector of corresponding items in the order they
; appear in s.
; (= (__ #(> % 5) #{1 3 6 8}) {false [1 3], true [6 8]})
; forbidden group-by
(fn [f s]
  ((fn step [ret f [x & xs]]
     (if x
       (let [k (f x)]
         (step (assoc ret k (conj (get ret k []) x)) f xs))
       ret))
    {} f (seq s)))
; the get function takes a default argument for when the key is not found,
; which is used to initialize a vector here. Note the use of seq for s, as
; the collection may be a set, where the [x & xs] destructering doesn't work.
; Intead of recursively going over a sequence, we can also use reduce:
(fn [f s]
  (reduce
    (fn [ret x]
      (let [k (f x)]
        (assoc ret k (conj (get ret k []) x))))
    {} s))

; 65: Write a function which takes a collection and returns one of :map, :set,
; :list, or :vector - describing the type of collection it was given.
; (= :map (__ {:a 1, :b 2}))
; forbidden: class, type, Class, vector?, sequential?, list?, seq?, map?, set?
; instance? getClass
(fn [coll]
  (let [x (rand-int 100) y (rand-int 100)
        p [x y] c (conj coll z)]
    (cond
      (= y (get c x)) :map
      (= p (get c p)) :set
      (= x (last (conj c x))) :vector
      :else :list)))
; we conj a random two element vector into the collection, map will treat it
; as a new key value pair, others treat it as a single item; set is a map too,
; so we can get the vector back with itself as the key; vector and list are
; differentiated by the position of the conj.

; 67: Write a function which returns the first x number of prime numbers.
(fn [x]
  (take x
        (remove
          (fn [n]
            (some #(= 0 (mod n %)) (range 2 (inc (int (Math/sqrt n))))))
          (iterate inc 2))))
; we just test each number n, each divided by numbers from 2 up to sqrt(n)

; 69: Write a function which takes a function f and a variable number of maps.
; Your function should return a map that consists of the rest of the maps
; conj-ed onto the first. If a key occurs in more than one map, the mapping(s)
; from the latter (left-to-right) should be combined with the mapping in the
; result by calling (f val-in-result val-in-latter)
; (= (__ - {1 10, 2 20} {1 3, 2 10, 3 15}) {1 7, 2 10, 3 15})
; forbidden: merge-with
(fn [f m & ms]
  (reduce
    (fn [ret x]
      (reduce
        (fn [r k]
          (conj r (if (r k) [k (f (r k) (x k))] (find x k))))
        ret (keys x)))
    (cons m ms)))
; note a map is a function itself, so (r k) and (x k) works

; 70: Write a function which splits a sentence up into a sorted list of words.
; Capitalization should not affect sort order and punctuation should be ignored
; (= (__ "Have a nice day.") ["a" "day" "Have" "nice"])
(fn [s]
(sort-by #(.toLowerCase %) (re-seq #"\w+" s)))

; 73: A tic-tac-toe board is represented by a two dimensional vector. X is
; represented by :x, O is represented by :o, and empty is represented by :e. A
; player wins by placing three Xs or three Os in a horizontal, vertical, or
; diagonal row. Write a function which analyzes a tic-tac-toe board and returns
; :x if X has won, :o if O has won, and nil if neither player has won.
; (= nil (__ [[:e :e :e]
            ;[:e :e :e]
            ;[:e :e :e]]))
;(= :x (__ [[:x :e :o]
           ;[:x :e :e]
           ;[:x :e :o]]))
(fn [board]
  (let [i [0 1 2]
        c (take 12 (cycle i))
        p (flatten (map #(repeat 3 %) i))
        zip #(map vector %1 %2)
        win? (fn [w]
               (some
                 (fn [x] (every? #(= w (get-in board %)) x))
                 (partition
                   3 (into (zip (into i p) c) (zip c (into (reverse i) p))))))]
    (cond
      (win? :x) :x
      (win? :o) :o)))
; we basically enumerate all possible winning positions, which fall into
; some regular patterns. I am sure there are better ways, but in the
; interest of time... Note the use of get-in to fetech value in a multiple
; dimensional vector: (get-in board [x y])

; 74: Given a string of comma separated integers, write a function which
; returns a new comma separated string that only contains the numbers
; which are perfect squares.
; (= (__ "4,5,6,7,8,9") "4,9")
(fn [s]
  (->> (re-seq #"\d+" s)
    (map #(Integer/parseInt %))
    (filter (fn [x]
              (let [r (int (Math/sqrt x))]
                (= x (* r r)))))
    (interpose ",")
    (apply str)))

; 75: Two numbers are coprime if their greatest common divisor equals 1.
; Euler's totient function f(x) is defined as the number of positive integers
; less than x which are coprime to x. The special case f(1) equals 1. Write a
; function which calculates Euler's totient function.
; (= (__ 10) (count '(1 3 7 9)) 4)
(fn [n]
  (->> (range 2 n)
    (filter (fn [x]
              (= 1 ((fn gcd [a b]
                      (if (= 0 b) a (gcd b (mod a b))))
                    x n))))
    count
    inc))

My solutions for the first 50 problems on 4clojure.com

2011-05-18T23:13:00-07:00

For someone without previous Lisp experience, the hardest part of learning Clojure programming seems to be the functional way of doing things. It is like math, one really needs to do some exercises in order to master it. At this point, 4clojure.com seems to be the best place for getting such exercises. It has a lot of problems for new clojurians to solve. These problems ask one to fill in the blank __ so the given expressions are true. To give a little challenge, some clojure built-in functions are forbidden to use for some problems. New problems are added from time to time on the site, so it surely can keep me entertained for a while.

I just finished the first 50 problems and think it might be helpful to post the solutions here. I tried to be functional and avoided using loops in the code. Some solutions are skipped as they seem trivial even for a functional newbie like myself. My solutions are probably just awful, but it is a fun experience nevertheless. I will post more solutions when I am done with them (Solutions No.50-75 and 76-100) Update: there are better solutions for problem 21, 27 and 44, contributed by visitors to the old site. But those comments are lost during system switch over.

; 21: Write a function which returns the Nth element from a sequence.
; (= (__ '(4 5 6 7) 2) 6)
; forbidden: nth
(fn [coll n]
((apply comp (cons first (repeat n rest))) coll))
; We first compose n rest functions to get progressively shorter lists

; till the desired element is the head, then take the head. A less

; fancy version just uses nthnext, but it feels like cheating:
(fn [coll n]
(first (nthnext coll n)))

; 22: Write a function which returns the total number of elements in

; a sequence.
; (= (__ '(1 2 3 3 1)) 5)
; forbidden: count
#(reduce + (map (fn [x] 1) %))
; We just turn each element into 1 and then add them up
; Note that (fn [x] 1) can be replaced by (constantly 1)

; 23: Write a function which reverses a sequence.
; (= (__ [1 2 3 4 5]) [5 4 3 2 1])
; forbidden: reverse
#(into () %)
; We exploit the property of the list, which alway add new element
; in front of the head. Also that the clojure sequences' equality
; evaluation is element based, so [1 2 3] equals to '(1 2 3)

; 26: Write a function which returns the first X fibonacci numbers.
; (= (__ 6) '(1 1 2 3 5 8))
(fn [x]
  (take x
    ((fn fib [a b]
        (cons a (lazy-seq (fib b (+ a b)))))
      1 1)))
; we first recursively construct a lazy sequence of infinite number of
; fibonacci numbers

; 27: Write a function which returns true if the given sequence is

; a palindrome.
; (true? (__ '(1 1 3 3 1 1)))
(fn [coll]
  (let [rc (reverse coll) n (count coll)]
    (every? identity
      (map #(= (nth coll %) (nth rc %)) (range (/ (dec n) 2))))))
; we naively compare half of the pairs of elment e(i) and e(n-i-1)

; 28: Write a function which flattens a sequence.
; (= (__ '((1 2) 3 [4 [5 6]])) '(1 2 3 4 5 6))
; forbidden: flatten
(fn flt [coll]
  (let [l (first coll) r (next coll)]
    (concat
      (if (sequential? l)
        (flt l)
        [l])
      (when (sequential? r)
        (flt r)))))
; we basically treat the nested collection as a tree and recursively

; walk the tree. Clojure's flatten use a tree-seq to walk the tree.

; 29: Write a function which takes a string and returns a new

; string containing only the capital letters.
; (= (__ "HeLlO, WoRlD!") "HLOWRD")
(fn [coll]
(apply str (filter #(Character/isUpperCase %) coll)))
; note the use of apply here, as str takes a number of args instead
; of a character collection

; 30: Write a function which removes consecutive duplicates from a sequence.
;  (= (apply str (__ "Leeeeeerrroyyy")) "Leroy")
(fn cmprs [coll]
  (when-let [[f & r] (seq coll)]
    (if (= f (first r))
      (cmprs r)
      (cons f (cmprs r)))))
; Basically a variant of the filter function. Note the sequence

; is destructed into first element f and the rest r.

; 31: Write a function which packs consecutive duplicates into sub-lists.
; (= (__ [1 1 2 1 1 1 3 3]) '((1 1) (2) (1 1 1) (3 3)))
(fn [coll]
  ((fn pack [res prev coll]
    (if-let [[f & r] (seq coll)]
      (if (= f (first prev))
        (pack res (conj prev f) r)
        (pack (conj res prev) [f] r)))
     (conj res prev))
    [] [(first coll)] (rest coll)))
; res is the final list, prev keeps the immediate previous sub-list.
; A much simpler version use partition-by:
#(partition-by identity %)

; 33: Write a function which replicates each element of a sequence

; n number of times.
; (= (__ [1 2 3] 2) '(1 1 2 2 3 3))
(fn [coll n]
(apply concat (map #(repeat n %) coll)))
; or more succintly:
(fn [coll n]
(mapcat #(repeat n %) coll))

; 34: Write a function which creates a list of all integers in a

; given range.
; (= (__ 1 4) '(1 2 3))
; forbidden: range
(fn [s e]
(take (- e s) (iterate inc s)))

; 38: Write a function which takes a variable number of parameters

; and returns the maximum value.
; forbidden: max, max-key
; (= (__ 1 8 3 4) 8)
(fn [x & xs]
(reduce #(if (< %1 %2) %2 %1) x xs))

; 39: Write a function which takes two sequences and returns the first

; item from each, then the second item from each, then the third, etc.
; (= (__ [1 2] [3 4 5 6]) '(1 3 2 4))
; forbidden: interleave
#(mapcat vector %1 %2)

; 40: Write a function which separates the items of a sequence by

; an arbitrary value.
; (= (__ 0 [1 2 3]) [1 0 2 0 3])
; forbidden: interpose
(fn [sep coll]
(drop-last (mapcat vector coll (repeat sep))))

; 41: Write a function which drops every Nth item from a sequence.
; (= (__ [1 2 3 4 5 6 7 8] 3) [1 2 4 5 7 8])
(fn [coll n]
  (flatten
    (concat
      (map #(drop-last %) (partition n coll))
      (take-last (rem (count coll) n) coll))))
; We partition the sequence, drop last one from each, then stitch them

; back take care the remaining elements too

; 42: Write a function which calculates factorials.
; (= (__ 5) 120)
(fn [n]
(apply * (range 1 (inc n))))
; clojure arithmetic functions can take a variable number of arguments

; 43: Write a function which reverses the interleave process into n

; number of subsequences.
; (= (__ [1 2 3 4 5 6] 2) '((1 3 5) (2 4 6)))
(fn [coll n]
(apply map list (partition n coll)))
; exploit map function's ability to take a variable number of

; collections as arguments

; 44: Write a function which can rotate a sequence in either direction.
; (= (__ 2 [1 2 3 4 5]) '(3 4 5 1 2))
; (= (__ -2 [1 2 3 4 5]) '(4 5 1 2 3))
(fn [n coll]
  (let [ntime (if (neg? n) (- n) n)
        lshift #(concat (rest %) [(first %)])
        rshift #(cons (last %) (drop-last %))]
    ((apply comp (repeat ntime (if (neg? n) rshift lshift))) coll)))

; 50: Write a function which takes a sequence consisting of items

; with different types and splits them up into a set of homogeneous

; sub-sequences. The internal order of each sub-sequence should be

; maintained, but the sub-sequences themselves can be returned in

; any order (this is why 'set' is used in the test cases).
; (= (set (__ [1 :a 2 :b 3 :c])) #{[1 2 3] [:a :b :c]})
#(vals (group-by type %))

Develop clojure Web applications with vim

2011-05-02T23:02:00-07:00

I recently started to learn clojure programming. It is an interesting experience. Ever since I learned computer programming almost 20 years ago, in Pascal, on a VAX minicomputer terminal, I have not experienced this newbie sensation with a computer language. The sense of excitement and novelty is high, and the eagerness to put the language to use is higher still. So for my new project at work, I am doing it with clojure.

This is a visual analytics project, and the visual part will be on the Web. It amazes me how much work has already been done for the Web using this 3 years old language. So it should be easy for me to get started. Here's what I have so far.

Know your lein

The standard build tool for clojure projects these days seems to be lein, short for leiningen. It is just a script, download it, make it executable, put it on your path. Also make sure you have java installed. Now go to whatever directory your project will live in, run lein:

lein new myproject

This creates the default project structure for a project creatively named "myproject", and downloads dependencies from the Internet, including clojure itself (clojure is built on java virtual machine, so the clojure language runtime is just a jar file).

Now in the "mypoject" directory, there is a "project.clj" file. This file controls everything about the project, except the writing code part. Edit the file according to your needs. There is a very extensive sample project.clj that I find very informative. My project.clj so far looks like this:

1 (defproject myproject "0.0.1-SNAPSHOT"
2   :description "MyProject has super duper visual analytics capabilities"
3   :dependencies [[org.clojure/clojure "1.2.1"]
4                  [org.clojure/clojure-contrib "1.2.0"]
5                  [compojure "0.6.2"]
6                  [hiccup "0.3.4"]
7                  [ring "0.3.7"]
8                  [commons-logging "1.1.1"]
9                  [org.apache.lucene/lucene-core "3.1.0"]
10                  [xalan "2.7.1"]
11                  [javaewah "0.1.0"]
12                  [com.my.work/secret-lib1 "0.3.4b"]
13                  [com.my.work/secret-lib2 "0.1.0"]]
14   :dev-dependencies [[lein-ring "0.4.0"]
15                      ;[org.clojars.autre/lein-vimclojure "1.0.0"]
16                      [clj-stacktrace "0.2.1"]]
17   :repositories
18             {"myrepo"
19              {:url
20               "https://myrepo.my.com:8080/artifactory/libs-release-local"}}
21   :source-path "src/clojure"
22   :java-source-path "src/java"
23   :warn-on-reflection true
24   ;:main com.my.myproject.runtime.core)
25   :ring {:handler com.my.myproject.runtime.core/app})

Let me explain line by line. The first line is the project name and the current version number (using the so called semantic versioning scheme).

The third line starts the dependencies definition. These dependencies are libraries written in jvm languages such as java or clojure. lein will automatically find and download them from public repositories if they are publicly available. This is the case for libraries referred in line 3 - 10, where the last three libs are open source java libs and the rest are open source clojure libs: compojure is a lightweight Web framework, which builds upon ring, which abstract HTTP into a simple API, hiccup allows one to write html in clojure syntax.

The lib on line 11 is also open source, however, it has not been packaged by the author and submitted to a pubic repository, so lein will not be able to find it. What I did was to package it as a jar file myself and deploy it to a private repository I setup for my team, so my team members can all access to the same libs without needing to commit the libs to our version control system, which is not suitable for handling binary data. This private repository is defined on line 17 to 20. Here the repository server is a standard installation of artifactory. The libs on line 12 and 13 are our in-house developed java libraries, which are deployed the same way.

By default, lein expect clojure code in "myproject/src". Since we will be mixing java code and clojure code, we put them in separate folders. These are defined in line 21 and 22.

Line 14 starts the dev-dependencies. These are the dependencies for developers' convenience and will not be included in the final product. lein-ring is a plugin for facilitating Web development in clojure that utilizes ring. Basically, it adds a ring command for lein. For example,

lein ring server-headless

will start a jetty server with the Web app running on port 3000 (default). The Web app is defined on line 25, which is just a simple app to show a greeting in this case. The code consists of two files. core.clj defines the main routing table for the Web app:

1 (ns com.my.myproject.runtime.core
2   (:use compojure.core
3         hiccup.middleware
4         com.my.myproject.runtime.views)
5   (:require [compojure.route :as route]
6             [compojure.handler :as handler]))
7
8 (defroutes main-routes
9   (GET "/" [] (index-page))
10   (route/not-found (page-404))
11   (route/resources "/"))
12
13 (def app
14   (-> (handler/site main-routes)
15       (wrap-base-url)))

views.clj defines the pages to show:

1 (ns com.my.myproject.runtime.views
2   (:use [hiccup core page-helpers]))
3
4 (defn index-page []
5   (html
6     [:head
7      [:title "Welcome"]
8      (include-css "/css/style.css")]
9     [:body
10      [:h1
11        "Hello World!"]]))
12
13 (defn page-404 []
14   (html
15     [:head
16      [:title "Sorry"]
17      (include-css "/css/style.css")]
18     [:body
19      [:h1 "Page not found"]]))

One very nice thing about lein-ring is that it will automatically pick up any changes made in the project. That's right, live changes, no need to wait for the code to compile, restart the server, etc, just refresh the browser and you will see the changes. This is extremely convenient for Web development, especially for experimentation in clojure REPL.

vimclojure

I am a vi addict. For my fix, there is a vimclojure plugin for clojure development with vim. To have dynamic features such as code snippet evaluation, code completion etc, there is a need to start a nailgun server so vimclojure can contact with a clojure REPL. This script is what I use:

1 # clojure jar is also installed in ~/.vim/lib
2 CL_CP=.:~/.vim/lib/*
3
4 if [ -f "project.clj" ]; then
5   CP=`lein classpath`:" $CL_CP"
6 else
7   CP=" $CL_CP"
8 fi
9
10 if [ $# -eq 0 ]; then
11      exec java -server -cp " $CP" vimclojure.nailgun.NGServer 127.0.0.1
12 else
13      exec java -server -cp " $CP" vimclojure.nailgun.NGServer $1
14 fi

I normally run this script in the root directory of the project, this allows "lein classpath" to pick up all the classpaths for the REPL session. There's also a lein-vimclojure plungin that will install vimclojure and start a nailgun server for you, but I found it does not load "user.clj", so my convenient functions defined there are not autoloaded. I will stick to my script.

For better navigation of clojure source code, vim users need TagList plugin. The plugin does not automatically work with clojure though. This blog post has a solution, and it worked for me. Basically, this tells TagList to treat clojure code just like other Lisp code, which it is.

That's all folks.

A conversation about the job talks of new Ph.D. graduates in computer science

2011-03-07T20:40:00-08:00

Yunyao and I work in a research center of a major technology company. At this time of a year, we attend job talks every week. These talks are technical presentations given by candidates for our research or development positions, as part of their two day job interviews here. Not surprisingly, the majority of the talks are given by newly graduated computer science PhDs who have published a lot of technical papers in premier computer science conferences. After a while, we kind of see some common deficiencies of these new graduates. Below is an instant messaging conversation we had right after attending a talk, which I hope would be helpful for some people.

YY: How do you like the talk?

HH: good slides

YY: yes, very well made

HH: lack of big picture in question handling

YY: but he seems to have some problem explaining the details (to outsider)

HH: I think he lacks big picture. Those questions could normally be answered by a few simple sentences

YY: yeah

HH: but he tried to explain the details, which further confused people. All the questions were about the big pictures, but he couldn't even understand what people were asking

YY: that's typical when the work is all about formulas, because he puts slides in such a way to help people understand, but it is not clear what the exact output his program would generate

HH: right

YY: it would be better that he first goes through actual input/output, and then explain the meaning of the output

HH: yeah, I am afraid they have not thought about that

YY: we are not interested in blackbox. For example, it is still unclear to me how he ... (omitted)

(a few exchanges on technical details are omitted)...

YY: I didn't ask too many questions, because I didn't want to make the situation worse for him

HH: right, I think he should have stepped back from the stories of his papers. Do not use the same ones in the job talk, instead, create a coherent new story for the job talk. It is the same problem with B (another candidate who gave talk here before).

YY: yes, and don't refer back to the paper. I am not going to read it.

HH: right, the talk should be self contained and coherent. The stories of the papers are normally not coherently fit to one another.

YY: No. Anyway, I am under-impressed.

HH: he just tried to repeat the stories of his papers, which are tenuous at best.

YY: yeah, I think the fact that his slides use simplified examples made things worse because it confuses people

HH: right. Anyway, I think it is a common problem with new graduates. They didn't seem to think too much about the problems they were solving. Everything is paper driven, and they just can't tell a story straight.

YY: exactly. This is because most people are not driven by problems, but by writing papers.

HH: right. They have a hunch, an idea, and they start writing a paper about it.

YY: (two smiles)

HH: I think this conference driven publishing culture in computer science contributes to this problem

YY: I think so

HH: well, these are the things beyond individual's control, so we can't really blame the student. If they are lucky, they get good adviser or mentors, then it is less of a problem...

YY: yeah, the student start from scratch, so it's really up to the adviser

HH: well then students often look up the publications of faculty members. If one faculty member publishes a lot, students will swarm to him instead of another one who publishes less (but maybe with higher quality). So it is a self perpetuating problem. Anyway, there's nothing we (in industry) can do about it.

YY: yeah.

Thinkpad Hotkeys in KDE4

2010-12-01T22:00:00-08:00

I have kept my Thinkpad X61 laptop up to date with Debian sid for a few years. The KDE4 in Debian sid is at version 4.4.5 at this moment, and I think it is mature enough for me to switch the laptop power management from my own hacked up solution to a KDE integrated one. My old solution still works for the most part, but there are some glitches after repeated supsend-resume cycles. As the system keeps evolving, I suspect more things would break.

Power management in KDE4 is handled by PowerDevil, which is disabled if powersaved is running (my old solution relied on powersaved). First I uninstalled powersaved and reboot, sure enough, all Fn hotkeys stops working. Now I go to KDE System Settings -> Advanced -> Power Management, and see the PowerDevil seems to be in a health state. The problem is that the hotkey presses are intercepted by ACPI so KDE does not receive these events. To stop the interruption, I edit /etc/modules as root and comment out thinkpad_acpi module, reboot. Now the Fn hotkeys should be registered in KDE, all we need to do is to make them do things we want.

To set up global hotkeys, we go to System Settings -> Input Actions, I add a new group called My Shortcuts and enabled it. Right-click My Shortcuts, New->Global Shortcut->DBus Command, I create a new entry Hibernate, and in the Action tab fill in needed dbus information for hibernating the system with PowerDevil (See screenshot). Basically, this is similar to issuing a console command "qdbus org.kde.powerdevil /modules/powerdevil suspend 4". Here, the parameter 4 is for hibernate (suspend to disk), 2 is for sleep (suspend to memory), and 1 for lock screen. In the Trigger tab, click the button, and press Fn+F12, notice that KDE recognizes this key as a Suspend key. Now click Apply, the Fn+F12 hotkey is setup. Now try Fn+F12, the system should hibernate. So far, the hibernate-resume cycles seem to be clean and problem free for me. I am using 2.6.36 kernel.

Overall, this seems to be easy enough. No scripts, no hacks.

Config Spring and DWR on Jetty with zero XML

2009-08-09T10:51:00-07:00

I hate xml configuation files. They look awful. As a result, they can be very time-consuming to write, and are very error-prone. There's no type-safty checking whatsoever. I would rather keep configurations within my Java source code. After all, we programmers are the only people looking at these configurations, why create trouble for ourselves? Luckily, with annotation support in Java, we can now completely do away with xml files for developing Web applications. Here is my recent experience integrating Spring framework with DWR on a Jetty sever.

Spring

Spring framework seems to be very popular in enterprise Java world. The core idea of "inversion of control (IoC)" seems to be trival (I bet any good programmers are already doing IoC without knowing the name). But sometimes, taking a simple idea seriously can get you a lot of mileage. Map-Reduce is another example on top of my head. Anyway, I decided to use Spring to manage objects in one of my dependency-rapidly-getting-out-of-hand Web projejcts. Spring was known for its xml hell, but recently advances have given it annotation-based configuration and JavaConfig. Now we can do Spring configuration completely in Java source code. To do this, I put these jars in my java built path:

spring.jar (Note that this is spring 2.5.6)
asm-3.2.jar
aspectj-1.6.5.jar
org.springframework.config.java-1.0.0.M4.jar
cglib-2.2.jar
dwr.jar
all jetty modules

JavaConfig needs at least one class annotated with @Configuation. Here's mine:

@Configuration
@AnnotationDrivenConfig
@ComponentScan("com.company.app")
public class AppConfig {
// no need to list our own beans here
// since we use autowiring and component scan
}

Yeah, the configuation file is pretty empty. In fact, @ComponentScan tells Spring to search for classes annotated with @Component (or several other stereotypes: @Controller, @Service, etc. See Spring doc for details) under "com.company.app" base package, and automatically inject all the dependencies labeled @Autowired. For example:

@Controller
@RemoteProxy(
  creator = SpringCreator.class,
  creatorParams =
    {
      @Param(name = "beanName", value = "UIController"),
      // this is needed due to a DWR bug
      @Param(name = "javascript", value = "UIController")
    },
  name = "UIController")
public class UIController {
  @Autowired
  private BackEnd backEnd;
  @Autowired
  private QueryProcessor queryProcessor;

// other stuff here
}

Both backEnd and queryProcessor will be automatically instantiated and injected here. This is really cool! What a time saver! Oh, don't forget to write setter for these private members. And the @RemoteProxy annotation is for DWR. More later.

Jetty

Now we can start our Web server this way too, without the damned web.xml and all that. Basically, we will embody a Jetty server in our Java application. Why Jetty? It's small, fast and flexible. I created a class to wrap it up and also created two servlets for it. One serves regular files, another handles ajax request using DWR.

@Component
public class UIServer {
  public static final File HTML_FILE_DIR = new File("../Web");
  private int port = 80;
  private Server jettyServer;

public void start() {
    try {
      // Create an instance of Jetty Web server
      jettyServer = new Server(port);
      ContextHandlerCollection contexts = new ContextHandlerCollection();
      jettyServer.setHandler(contexts);

// this servlet serves static files
      ServletContextHandler ctxDocs=
      new ServletContextHandler(contexts, "/", ServletContextHandler.SESSIONS);
      ctxDocs.setResourceBase(HTML_FILE_DIR.toString());
      ServletHolder ctxDocHolder= new ServletHolder();
      ctxDocHolder.setInitParameter("dirAllowed", "false");
      ctxDocHolder.setServlet(new DefaultServlet());
      ctxDocs.addServlet(ctxDocHolder, "/*");

// this DWR servlet handles UI requests
      ServletContextHandler ctxUI =
      new ServletContextHandler(contexts, "/ui", ServletContextHandler.SESSIONS );
      ServletHolder ctxUIHolder= new ServletHolder();
      ctxUIHolder.setInitParameter("debug", "true");
      ctxUIHolder.setInitParameter("jsonpEnabled", "true");
      ctxUIHolder.setInitParameter("crossDomainSessionSecurity", "false");

// Specify the classes (comma delimited fully qualified class names)
      // to be exposed to Web browser
      ctxUIHolder.setInitParameter("classes", "com.company.app.UIController");
      ctxUIHolder.setServlet(new DwrServlet());

ctxUI.addServlet(ctxUIHolder, "/*");
contexts.setHandlers(new Handler[]{ctxUI, ctxDocs});

jettyServer.start();
jettyServer.join();

} catch (Exception e) {
e.printStackTrace();
}
}

We now need to bootstrap our application with JavaConfig by creating an application context.

public class App {

public static void main(String[] args) throws Exception {
JavaConfigApplicationContext ctx =
new JavaConfigApplicationContext(MidasConfig.class);

// so DWR knows where to find classes
SpringCreator.setOverrideBeanFactory(ctx);

UIServer uiServer = ctx.getBean(UIServer.class);
uiServer.start();
}
}

DWR

DWR allows javascript on browser to directly call Java methods on server. It's like a RPC thing, which is pretty convenient. It's my favirate ajax communication layer. Notice we have already covered most of the configurations needed for DWR in the code fragments above. Here's some explanations. The @RemoteProxy annotation basically says that "expose this class to javascript". Within such classes, @RemoteMethod annotation makes annotated methods visible for javascript to call. More details please see DWR documentation. In addition to such annotations, we did two tricks here to make DWR configuration completely devoid of any xml files. First we passed DWR some configurations through jetty's servletholder. Second, we used SpringCreator of DWR, which basically ask Spring for objects.

Well, it took me lots of googling to put this together. Hope it would be useful for someone.

Tails & No Tales Cat Show

2009-01-04T07:37:00-08:00

This is the first time we have even been to a cat show. We have enjoyed it a lot!

The show cats are all very tidy and cute. Too bad that spectaculors like me cannot touch or pet any of them. :(

But I still love my kitties adopted from shelter the most. :) Even though they could not come to the show (well, even if they were allowed to, I really doubt that they would enjoy it), they liked the peacock feather we bought for them very much.

Here are some photos taken today.

A cat looks like Yoda.

Notice the curly fur.

Notice the tail

The cat loved the feather so much that he jumped out of the stage to get it.

Cas without fur is not my cup of tea, but she seems to be very sweet.

Want a hug ...

A beautiful red cat.

Curly ears

A sleeping beauty

Pose for a picture

Relaxing during the break.

Seasonal Greetings from the Cats

2008-12-31T22:20:00-08:00

Tom is waiting for Santa ...

Mayori want to be Santa's little helper ...

Thanksgiving Dinner

2008-12-06T18:02:00-08:00

I have been waiting to try the Tofurkey for a few weeks ever since I saw the ad from Trader Joe about it. So I finally tried out on Thanksgiving.

This is what Tofurkey looks like out of the box. Then I cooked it in the oven. It tastes better than I expected, but a little too much for two people.

Wedding Gift from Huahai's Mom

2008-11-14T03:57:00-08:00

Finally, after our trip to China this year, we got our wedding gifts from Huahai's parents --- a special Olympic umbrella. 😛

Winking Tom & Superman

2008-06-15T06:37:00-07:00

"Finding happiness: a cat's story"

2008-06-06T20:56:00-07:00

Long long time ago, there's a prince named Thomas. He's a handsome boy, but often sad, because he's parents passed away when he's very young.

很久很久以前，有一个年轻英俊的王子叫作唐唐。他名如其人，长得仪表唐唐。可是因为从小就父母双亡的缘故，他的心里总是充满了淡淡的忧伤。

One day, Thomas prays to Godess Felines: dear goddess, I have been good all these years. Please give me something that will bring me happiness.

终于有一天，唐唐王子鼓起勇气向猫神巴思特祈祷道：＂亲爱的巴思特女神，这么多年以来我一直都努力做一个合格的猫王子，可是我并不快乐。您可以用您的魔力赐与我快乐吗？

Goddess Felines says: I know you are a good boy, but happiness is not something handed out by gods. You have to find your own happiness.

Where can I find my happiness, asks Thomas?

Search in the basket of life, you will find it, answered Godess Felines.

巴思特女神说"唐唐，我知道你是一个好孩子。可是幸福不是神仙可以赐与的。你必须自己寻找到幸福。"

＂那么我在哪里可以找到幸福呢？＂唐唐急切的问道。

＂它在生命的篮子里＂巴思特女神回答道。

Basket of life? Thomas is puzzled: "I have a basket, all my toys are in it. Is that basket of life?". Godess Felines smiles and leaves. Disappointed, Thomas goes to his basket, and takes out his favirate toy, a green shining ball.

"生命的篮子？"唐唐困惑的问道。"我倒确实有一个篮子，里面放满了我的玩具。那个是生命的篮子吗？"

可是巴思特女神什么也没说便微笑着消失掉了。

唐唐很失望的走到玩具篮子之前，取出平时最心爱的闪光的小绿球。

Thomas carries the shining ball to play ground.

唐唐叼着闪光的小绿球来到游乐场。

Thomas plays with the shining ball upside down.

他把小球抛到半空中然后接住。

He plays the ball downside up.

接着再前空翻后滚翻。

Thomas spins with the ball.

唐唐抱着小球翻滚得越来越快，最后只见他灰色的影子在快速地移动。

Thomas is tired, but is still not happy.

玩了很久以后，唐唐觉得有些累了，可是却没有觉得开心。

Unbekown to Thomas, a wild wild cat is watching him in the shadow all the time.

但唐唐却万万没有想到，就在他玩耍的时候，不远的地方有一双蓝色的眼睛一直在暗地里观察着他的一举一动。

Seeing Thomas is tired and unable to defend himself, the wild cat rushs out of thedarkness, and goes for the shinning ball!

Kitty Singing a Birdy Song

2008-06-03T23:42:00-07:00

偶家的娇娇爱唱歌

How to save millions per year for your newspaper?

2008-04-20T07:45:00-07:00

Dear CEO of XXX News:

I am writing in regard to an automatic news article writer (NAW) that can ** **save your company millions of dolloar per year (see appendix below for peudo code). The basic idea of NAW is to be able to automatically generate news articles with comparable quality to news articles written by actual human reporters by your company.

As a proof of concept, I have conducted a comprehensive comparison study. The results show that on issues with regard to Olympic Torch Relay 2008, the similarity of the automatically generated news articles and those manually written by human reporters is higher than 99%, making them undistinguishable by any of your valuable readers. With a click of a button, you can generate news articles using NAW at your wish anytime anywhere. More importantly, your readers will not realize the differences.

Since issues with regard to the Olymipic Torch Relay 2008 refect most if not all the issues about China, the above result safely indicates that NAW can be used to replace your news reporters on any issues with regard to China, thus allowing them to concentrate on more important issues such as Iraq, human right issues in US inner cities, independent requests from Hawaii, Porto Rico and Alaska, and so on.

Assume that your news paper needs one report with regard to China about every other day. That is about 180 news articles each year. Assume the cost of writing each news articles, including transportation, lodging, meals, salary for your news reporter and his/her crew members, is merely $8000. By investing on this software, with a fix cost of $500 and no operating cost (since it can work on any of the existing computers of your company), you can save nearly $1.5 millions per year from now on!

More importantly, with a little more input from human reporters, NAW can be adopted to report issues on many other countries. That is again over millions of saving each year!

Thank you very much for your time. Please do let me know should you have any questions. I am looking forward to discuss you next steps.

Best regards,

an inventor

Appendix:

Input:   

$event:      event name
$location:   event location
$date:       event date
           

Output: 

$news:       new article

 
Function:

NewsArticleWriter (event, location, date)

  Begin:

    Let $numProtester = a random number between 3000 to 10000

    Let $numSupporter = numProester/10

    Let $issues = {"Tibet", "FLG", "AgainstChinaMade", "HumanRightIssues", "Free $ProvinceName", $AnyOtherCurrentHotIssue}

    Let $typesProtesters= a random subset from 

    Let $typesSupporters =  {aggressive males bused in and paid by Chinese consolate or by Pro-China organization}

    Let $name = a random popular Chinese name

    Let $locations = an array of nearby locations of $location


    news +=  Write(event, location, date)

    news +=  Write(event, typesProtesters)

    news +=  Write(numProtester, numSupporter)

    // a function writing elaborated version on 
    // predefined topics for each types of protesters
    news +=  WriteMore(typesProtesters) 
   

    for $i between 1 and lengthOf($locations)
       news += Write($location[i], $typesProtesters)

    news +=  Write(name, typesSupporters)

  End

Suspending to RAM on Debian laptop

2008-04-17T23:39:00-07:00

I am tracking Debian sid on my Thinkpad laptop, a few months ago it started to use pm-utils and broke suspending to RAM (sleep) functionality. Basically, the machine would go to sleep then immediately resume.
It turned out that this problem can be easily fixed by creating a file /etc/pm/config.d/local, and put in a line

SUSPEND_MODULES="e1000"

The reason is that pm-utils by default does not unload Ethernet card module e1000, so the machine would be waken up by Ethernet card activities.

YouTube censors BBC's documentary in USA

2008-03-10T15:15:00-07:00

BBC is broadcasting a documentary series titled "A year in Tibet". From the program description on BBC Web site, this documentary shows the real life today in Tibet. The first segment was shown on Mar. 6. Not surprisingly, there is a Youtube video of this first show. Surprisingly, if you attempt to access this video in US, Youtube says "This video is not available in your country." Interestingly, people in UK have confirmed that this video is accessible in UK!

Obviously, this block of access could not have been requested by BBC on the basis of copyright violation. Or how could you explain the availability of the video on Youtube for UK visitors? One can only speculate on who is behind this blockage. The Free Tibet people? What are they afraid of? Shouldn't showing the reality of Tibet today further their cause? The Chinese government? Since when Youtube is afraid of Chinese government? If Youtube is afraid of them, how to explain the abundant Free Tibet videos on Youtube?

One thing is clear, people in US are now proved to be actively and specifically targeted for information blockage. I thought only China has the Great Firewall. Where's the outrage?

Update 03/12/2008: if you are looking for confirmation, check out the comments on digg

Update 03/17/2008: some people point out that it is BBC's policy to not allow access to their video outside UK. One comment on Slashdot pointed out that the video is blocked in Netherland as well. If these are true, my speculation could be wrong.

However, it is rather depressing to see that some of the comments to my post display an unabated bias and contempt towards people of my heritage. For example, the comment below accusing me of justifying totalitarianism is so unwarranted to the point of absurdity, especially considering that some of my posts here would be considered subversive to party rule.

Another comment on Slashdot put it in plain words: Identifying yourself as Han Chinese has already shown how biased you are so no more needs to be said. I simply cannot understand the reasoning behind this statement. Why would identifying one's own ethnic identity automatically disqualify one from being listened to? I hope this is another cultural gap that I have yet to overcome, is it? Can any one enlighten me a bit on this? Anyway, I hope these people are not representative of the general population here in the West.

an interesting birthday gift

2008-03-02T22:34:00-08:00

Huahai has received his birthday gift from Yunyao, a few days early.

What's inside?

Wow, a scenic flight certificate, what a lovely gift! Thank you, babe.