Finish JSON

Somewhere right now, a server is sending your phone a page of text. Curly brackets, quote marks, commas. Your weather app reads it in a blink and shows you 14 degrees and rain. And if one comma is missing, a good app doesn't guess. It tells you where. At the end of 1.7, your library couldn't do any of that. It could build a tree by hand and print it. It couldn't read a single letter.

1.7 ended with two aches. The weather app's reply was a String, and nothing in your library could turn it into a tree. And sending a Person as JSON meant a with chain by hand, for every class you ever write.

This chapter fixes both, in two more stages, the way 0.7 built Wordle. Every stage runs before the next one starts. Stage 3 reads JSON text and builds the tree, and 1.5's exceptions get their real job: saying where broken text went wrong. Stage 4 turns your own classes into JSON and back, with 1.6's reflection. Then we run every test the library has, all 186, and look back at Phase 1.

Fair warning about three things. You'll hand the parser 100,000 open brackets, to try to knock it over. You'll watch it name the wrong line for a mistake. And you'll send the number 9007199254740993 out as JSON and get a different number back, with no error at all.

What you'll be able to do

• Cut JSON text into tokens, each one knowing the line and column it came from.
• Read a recursive-descent parser, and watch the call stack grow and shrink with the brackets.
• Read a JSON file, and when it's broken, explain why the error names the line it does.
• Turn your own classes into JSON and back with one line each, and name six places where that goes wrong.

Stage 3Text in: a parser that names the line

A quick recap from 1.5. A method that can't do its job throws an exception, which falls down the plates until a catch stops it. Exceptions under RuntimeException are unchecked, so javac never forces anyone to plan for them. IOException is checked, because a disk can fail however careful your code is. And your own ScoreFileException kept its line number as a field with a getter.

Stage 3 spends all of that. Reading text and building the tree from it is called parsing. The code that does it is a parser. Ours will read text from files, from the internet, typed by people. Much of it will be wrong: a missing comma, a stray bracket, a word spelt ture.

So Stage 3 comes in four goes, and each one runs. In 3a we decide what the parser does with bad text. In 3b we cut text into words. In 3c we fit the words together into a tree. In 3d we read a real file, and break it.

3a — Decide what bad text throws

When the text is wrong, the parser throws. But throws what?

ScoreFileException had a line number. A JSON file needs one more number, because a single line can be long. A column is how far along the line you are, counting from 1, like a seat number in a row. With a line and a column, you can point at one exact character.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19public class JsonParseException extends RuntimeException {

    private final int line;
    private final int column;

    public JsonParseException(String problem, int line, int column) {
        super(problem + " at line " + line + ", column " + column);
        this.line = line;
        this.column = column;
    }

    public int line() {
        return line;
    }

    public int column() {
        return column;
    }
}

Line 1 is the decision: extends RuntimeException. That makes it unchecked, for every caller. Lines 3 and 4 are two private final fields, as in 1.2's Money: where the problem was found.

Line 7 hands the message to the parent with super, as ScoreFileException did. It reads like unterminated string at line 1, column 5. There's no cause this time. The parser is where the bad news starts, so there's no earlier exception to keep.

Lines 8 and 9 fill the fields. Lines 12 to 18 are two getters, so a program can ask for the numbers. In 3d, one draws a ^ under the exact spot.

Why unchecked? For the reason 1.5 gave: forcing a try around every parse, even of text you trust, is how empty catch blocks get written. Gson makes the same choice. The cost is real, though. For text from a stranger, javac won't remind you to catch it.

The rest of Stage 3 builds Json.parse, which turns text into a tree. It's already in src/stage3, so you can watch the decision first and open the machine afterwards. NoCatchNeeded.java parses two pieces of text, with no try and no throws. The second has a comma with nothing after it, which JSON forbids.

1
2System.out.println(Json.parse("[1, 2]"));
System.out.println(Json.parse("[1, 2,]"));

$ cd stage3
$ java NoCatchNeeded
[1,2]
Exception in thread "main" JsonParseException: expected a value but found ']' at line 1, column 7
	at JsonParser.error(JsonParser.java:137)
	at JsonParser.parseValue(JsonParser.java:58)
	at JsonParser.parseArray(JsonParser.java:80)
	at JsonParser.parseValue(JsonParser.java:41)
	at JsonParser.parseDocument(JsonParser.java:28)
	at Json.parse(Json.java:21)
	at NoCatchNeeded.main(NoCatchNeeded.java:7)

3b — Cut the text into words

Read this sentence: the cat sat. You didn't see eleven letters and two spaces. You saw three words. Your eyes cut the letters into words first. Only then did your brain check that the words made a sentence.

The parser works the same way, in two parts. A tokenizer cuts the text into words, and one word of JSON is called a token. Then the parser proper checks that the words fit together.

JSON has very few kinds of word. Six bits of punctuation: { } [ ] : ,. Strings in double quotes. Numbers. And true, false and null. Our tokenizer adds one made-up kind, end, meaning "nothing left". Spaces and new lines aren't tokens at all. They only sit between them.

Each token is a small object. Here's its class, minus one method we'll come to.

1
2
3
4
5
6
7
8
9
10
11
12
13public final class Token {

    public final String kind;
    public final String text;
    public final int line;
    public final int column;

    public Token(String kind, String text, int line, int column) {
        this.kind = kind;
        this.text = text;
        this.line = line;
        this.column = column;
    }

Lines 3 to 6 are four fields. kind is what sort of token it is, as text: "{", "string", "number", "end" and so on. text is what it said, like a string's letters or a number's digits. line and column are where it started.

They're public final, not private with getters as in 1.2. That's safe here. Nobody can change a final field after the constructor, so there's nothing to protect. Lines 8 to 13 are the constructor, which fills all four, once.

The method we skipped is describe(). It turns a token into words for an error message, like string "born" or '{'.

Inside next() — the two blanks are yours

next() is the method that does the cutting. From here on, Stage 3 works the way 0.7's Stage 3 did. I give you the shape. The lines that actually decide the answer are left blank, and those are yours.

The tokenizer keeps three numbers as it goes. pos is which slot of the text it's looking at, counting from 0. line and column say where that slot is, for humans, counting from 1.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27public Token next() {
    skipWhitespace();
    int startLine = line;
    int startColumn = column;
    if (pos >= text.length()) {
        return new Token("end", "", startLine, startColumn);
    }
    char c = text.charAt(pos);
    if (c == '{' || c == '}' || c == '[' || c == ']' || c == ':' || c == ',') {
        advance();
        return new Token("" + c, "" + c, startLine, startColumn);
    }
    if (/* BLANK 1 — does a string start here? */) {
        return new Token("string", readString(), startLine, startColumn);
    }
    if (/* BLANK 2 — does a number start here? */) {
        return new Token("number", readNumber(), startLine, startColumn);
    }
    if (c >= 'a' && c <= 'z') {
        String word = readWord();
        if (word.equals("true") || word.equals("false") || word.equals("null")) {
            return new Token(word, word, startLine, startColumn);
        }
        throw new JsonParseException("unknown word '" + word + "'", startLine, startColumn);
    }
    throw error("unexpected character '" + c + "'");
}

Line 2 skips spaces, tabs and new lines. Lines 3 and 4 note where the token starts. Lines 5 to 7: no text left? Hand back the end token. Line 8 looks at the current character, and the rest is a chain of questions about it.

Lines 9 to 12: one of the six punctuation marks? Step past it, and hand back a token whose kind is that mark. "" + c glues a char onto an empty String, making a one-letter String.

Lines 14 and 17 are given. Once you know a string or a number starts here, readString() or readNumber() reads the rest of it.

Lines 19 to 25: a small letter starts a word. true, false or null is a token. Anything else, like ture, is refused. Line 26 refuses any other character, like a +, with a tiny helper that builds the exception at the current position.

Blank one decides whether a string starts here. Every JSON string starts with the same character. Compare c with it using ==, the way line 9 does. A char goes in single quotes, so a double quote mark is written '"'.

Blank two decides whether a number starts here. A JSON number starts with a minus sign or a digit. That's two tests joined by ||, 0.3's "or". The class has a helper for the second test, isDigit(c), which is true for '0' to '9'. Get this one wrong and 1815 comes back as "unexpected character '1'".

Here's TokenDump, the program that tests your two lines.

1
2
3
4
5
6
7
8String text = "{\"born\": 1815,\n \"langs\": [\"en\"]}";
JsonTokenizer tokenizer = new JsonTokenizer(text);
Token t = tokenizer.next();
while (t.kind.equals("end") == false) {
    System.out.println(t.line + ":" + t.column + "  " + t.describe());
    t = tokenizer.next();
}
System.out.println(t.line + ":" + t.column + "  " + t.describe());

Line 1 is the JSON, inside a Java String. Every \" is a quote mark, and \n is a new line, both from 0.6. So the text is two lines long, and the second starts with a space.

Line 2 makes a tokenizer for it. Line 3 asks for the first token with next(), your next(). Lines 4 to 7 loop until the end token, printing where each token started and its description. Line 8 prints the end token too.

$ cd stage3
$ java TokenDump
1:1  '{'
1:2  string "born"
1:8  ':'
1:10  number 1815
1:14  ','
2:2  string "langs"
2:9  ':'
2:11  '['
2:12  string "en"
2:16  ']'
2:17  '}'
2:18  the end of the text

Every step forward goes through one small method, and it's the reason the positions are always right.

1
2
3
4
5
6
7
8
9
10
11private char advance() {
    char c = text.charAt(pos);
    pos++;
    if (c == '\n') {
        line++;
        column = 1;
    } else {
        column++;
    }
    return c;
}

It moves pos on by one, and hands back the character it stepped over. A new line also moves to the next line and back to column 1. Anything else moves one column along. readString, readNumber and readWord are loops that call advance() until their token ends. Each throws when a token is broken inside. A few of their messages, all checked by the tests:

unterminated string at line 1, column 5                   <- "abc   with no closing quote
no leading zeros in JSON numbers at line 1, column 2      <- 01
unknown word 'tru' at line 1, column 1                    <- tru
unexpected character ''' at line 1, column 1              <- 'hi'   JSON only allows "

3c — A parser that calls itself

Here's how JSON describes itself, in plain words.

• A value is one of these: an object, an array, a string, a number, true, false or null.
• An array is a [, then values separated by commas, then a ].
• An object is a {, then key: value pairs separated by commas, then a }. Each key is a string.

Read the second rule again. An array is made of values, and a value can be an array. The rule uses itself.

So the code copies the rules, one method per rule. parseArray calls parseValue for each item. And parseValue, if it sees a [, calls parseArray. The methods call each other as deep as the brackets go.

That's recursion, from 0.4's countdown, spread across two methods. The base case is a value with no insides, like a string or a number, which parseValue builds without calling anything. This kind of parser is called recursive descent. It descends into each bracket by calling itself.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21import java.util.ArrayList;
import java.util.LinkedHashMap;

public static final int MAX_DEPTH = 500;

private final JsonTokenizer tokenizer;
private Token current;
private int depth = 0;

public JsonParser(String text) {
    this.tokenizer = new JsonTokenizer(text);
    this.current = tokenizer.next();
}

public JsonValue parseDocument() {
    JsonValue value = parseValue();
    if (current.kind.equals("end") == false) {
        throw error("expected the end of the text", current);
    }
    return value;
}

Lines 1 and 2 import the ArrayList and LinkedHashMap from 1.7, to hold an array's items and an object's pairs. Line 4 is one limit for the whole class, as static final meant in 1.2. It caps how deep the parser will go, and at the end of 3c it stops an attack.

Lines 6 to 8 are the parser's state. A tokenizer. current, the token it's looking at, because the parser only ever looks one token ahead. And depth, how many brackets deep it is. Lines 10 to 13, the constructor, read the first token into current.

Lines 15 to 21 are where parsing starts. Parse one value, which might be a whole tree, then check nothing is left over. [1] [2] is two values, and JSON allows one, so that throws.

Two tiny helpers sit at the bottom of the file. advance() is current = tokenizer.next();. error(expected, found) builds the "expected … but found …" message you saw in NoCatchNeeded.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26private JsonValue parseValue() {
    Token t = current;
    switch (t.kind) {
        case "{":
            return parseObject();
        case "[":
            return parseArray();
        case "string":
            advance();
            return new JsonString(t.text);
        case "number":
            advance();
            return number(t);
        case "true":
            advance();
            return new JsonBoolean(true);
        case "false":
            advance();
            return new JsonBoolean(false);
        case "null":
            advance();
            return JsonNull.INSTANCE;
        default:
            throw error("expected a value", t);
    }
}

This is the first rule, a value is one of these, as a switch from 0.3. A switch on Strings compares with .equals, so 1.1's == trap can't bite. Each case ends with a return or a throw, so no break is needed.

Lines 4 to 7 are the values with insides. { and [ hand the job to parseObject or parseArray.

Lines 8 to 22 are the five base cases. Each calls advance() to move past its token, then builds the matching value from 1.7. Numbers go through a helper, number, which also refuses one too big for a double.

Lines 23 and 24 catch anything else, like a ] where a value should be. That's the throw in NoCatchNeeded.

Now the method with the recursive call in it. After each item, parseArray has to decide: is another item coming, or is the array finished? Those two decisions are yours.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22private JsonArray parseArray() {
    enter();
    advance();                                   // the [
    ArrayList<JsonValue> items = new ArrayList<>();
    if (current.kind.equals("]")) {
        advance();
        depth--;
        return new JsonArray(items);
    }
    while (true) {
        items.add(parseValue());
        if (/* BLANK 3 — is another item coming? */) {
            advance();
        } else if (/* BLANK 4 — is the array finished? */) {
            advance();
            depth--;
            return new JsonArray(items);
        } else {
            throw error("expected ',' or ']'", current);
        }
    }
}

Line 2, enter(), adds one to depth, for a reason at the end of 3c. Line 3 steps past the [. Line 4 makes an empty list. Lines 5 to 9 handle []: step past the ], take one off depth, hand back an empty array.

Line 10 is while (true). Its condition never turns false, so the loop only ends when something inside leaves it. Here every way out is a return or a throw.

Line 11 is the recursive call: items.add(parseValue()). Parse one value and add it to the list. It might be a number. It might be a whole array with arrays inside it. parseArray doesn't know and doesn't care.

After it, current is whatever token follows the item. Blank three asks whether that token is a comma. Line 5 shows how to ask what kind the current token is. Blank four asks whether the array is finished, which is the same question with a different mark. Anything else falls through to line 19, the error you'd get for [1 2].

parseObject is the same shape with { and }. Its loop expects a string key, a :, then calls parseValue. Each pair goes into a LinkedHashMap, so if a key appears twice, the second one wins.

Now watch it run on [1, [2]]. Count the plates on the left, and count the open brackets on the right.

That's what the NoCatchNeeded trace was showing you: parseValue, parseArray, parseValue, parseDocument. Those are the plates of step 3, from a different input. Java printed them at the moment [1, 2,] went wrong. The height of the spike is the depth of the brackets. That's the whole idea of recursive descent.

100,000 open brackets

Here's the attack. Every open bracket puts two more plates on the spike, and 0.4 showed you the spike has a fixed height. So what if somebody sends this parser 100,000 [ in a row? That's not a silly question. A program that reads JSON from the internet reads text from strangers. Some send the worst text they can think of, on purpose.

1
2
3
4
5
6
7
8
9StringBuilder text = new StringBuilder();
for (int i = 0; i < 100000; i++) {
    text.append('[');
}
try {
    Json.parse(text.toString());
} catch (JsonParseException e) {
    System.out.println("JsonParseException: " + e.getMessage());
}

Lines 1 to 4 build the text with a StringBuilder from 0.6, adding one [ a hundred thousand times. Lines 5 to 9 parse it, inside a try that catches our exception and prints its message.

$ cd stage3
$ java DeepBrackets
JsonParseException: nested deeper than 500 at line 1, column 501

1
2
3
4
5
6private void enter() {
    depth++;
    if (depth > MAX_DEPTH) {
        throw new JsonParseException("nested deeper than " + MAX_DEPTH, current.line, current.column);
    }
}

Every parseArray and parseObject calls enter() first. It adds one to depth. Over 500, it throws, long before the spike could run out. Every way out of those methods takes one off again, as the stepper showed. Five hundred deep is far more than any honest JSON needs.

Want to see what the limit protects you from? In stage3/JsonParser.java, change MAX_DEPTH = 500 to MAX_DEPTH = 1000000. Compile again, and run DeepBrackets.

$ cd stage3
$ javac *.java
$ java DeepBrackets
Exception in thread "main" java.lang.StackOverflowError
… a few lines inside the tokenizer, then this pair, over and over:
	at JsonParser.parseValue(JsonParser.java:41)
	at JsonParser.parseArray(JsonParser.java:80)
… about a thousand lines of it. I trimmed them.

There it is: 0.4's runaway recursion, from honest code and hostile text. And notice that DeepBrackets' catch didn't catch it. A StackOverflowError is an Error, not a JsonParseException, so it fell straight past, out of main.

Put MAX_DEPTH back to 500 when you're done. Stage 4 starts from these exact files.

3d — Read a file, and point at the mistake

The last piece of Stage 3 reads JSON from a file. It's the other method in the library's front door, Json.java.

1
2
3
4
5
6
7
8
9
10
11
12public static JsonValue parseFile(Path path) throws IOException {
    StringBuilder text = new StringBuilder();
    try (BufferedReader reader = Files.newBufferedReader(path)) {
        String line = reader.readLine();
        while (line != null) {
            text.append(line);
            text.append('\n');
            line = reader.readLine();
        }
    }
    return parse(text.toString());
}

Line 1 declares throws IOException, the library's first checked exception. A file can go missing however good our code is, so callers must plan for it.

Line 2 makes a StringBuilder to collect the text. Line 3 opens the file with 1.5's try-with-resources, so it's closed whatever happens. Lines 4 to 9 read it line by line. readLine() strips off each line's new-line character, so line 7, text.append('\n'), puts it back. Without it, every error would say line 1.

Line 11 parses the text. A JsonParseException from parse falls straight out of parseFile too, with no declaring, because it's unchecked.

Here's the file it reads: Ada from 1.7, as text this time.

$ cd stage3
$ cat ada.json
{
  "name": "Ada Lovelace",
  "born": 1815,
  "languages": ["English", "French"],
  "address": { "city": "London", "postcode": null },
  "note": "She wrote \"the first program\"."
}

And here's the demo that uses all of Stage 3.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19String file = "ada.json";
if (args.length > 0) {
    file = args[0];
}
Path path = Path.of(file);

try {
    JsonObject ada = (JsonObject) Json.parseFile(path);
    System.out.println(ada.toJson());
    System.out.println(ada.getString("name") + " lived in "
            + ada.getObject("address").getString("city") + ".");
} catch (NoSuchFileException e) {
    System.out.println("There is no file called " + file + ".");
} catch (JsonParseException e) {
    System.out.println(file + ": " + e.getMessage());
    List<String> lines = Files.readAllLines(path);
    System.out.println("  " + lines.get(e.line() - 1));
    System.out.println("  " + spaces(e.column() - 1) + "^");
}

Lines 1 to 5 pick the file: ada.json, unless you name another, like 0.7's practice mode.

Line 8 parses the file. parseFile hands back a plain JsonValue. We know this one is an object, so (JsonObject) is a cast, as in 1.2. Line 9 prints the tree. Lines 10 and 11 use 1.7's typed getters to dig out the name and the city.

Two catches, one per kind of bad news. A missing file gets a sentence. A parse error gets the message, then a picture of where.

For the picture, line 16 uses a method 1.5 didn't need. Files.readAllLines(path) reads a whole file at once and hands back a List<String>, one item per line. Line 17, lines.get(e.line() - 1), takes the broken line, with - 1 because lists count from 0. Line 18 prints spaces(e.column() - 1) and then a ^. spaces(n) is a four-line helper at the bottom of JsonDemo.java: a loop that glues n spaces into one String. That's why the column was worth keeping as a number.

$ cd stage3
$ java JsonDemo
{"name":"Ada Lovelace","born":1815,"languages":["English","French"],"address":
{"city":"London","postcode":null},"note":"She wrote \"the first program\"."}
Ada Lovelace lived in London.

From text in a file to a tree you can ask questions of. Even the \" inside the note survived the trip both ways.

The missing comma

Now the broken file. Look closely at the end of its third line.

$ cd stage3
$ cat broken.json
{
  "name": "Ada Lovelace",
  "born": 1815
  "languages": ["English", "French"]
}

There's no comma after 1815. The mistake is at the end of line 3, right after the number. So the error will say line 3. Won't it?

$ cd stage3
$ java JsonDemo broken.json
broken.json: expected ',' or '}' but found string "languages" at line 4, column 3
    "languages": ["English", "French"]
    ^

So when an error points at a line that looks fine, look at the line before it. That habit works far beyond JSON.

And the last case, a file that isn't there at all:

$ cd stage3
$ java JsonDemo nope.json
There is no file called nope.json.

Stage 3, tested

Your copy already passed. Here's the library itself, compiled from scratch.

$ cd stage3
$ javac *.java
$ java JsonTests
140 passed, 0 failed

javac printed nothing, which is what success looks like. Then 140 checks, up from 1.7's 69. They cover nesting, the depth limit, and every error message on this page, line and column included. A new group, testRefusals, also goes back over Stages 1 and 2. It checks that each refusal really refuses, with exactly the right message.

Stage 4Objects in, objects out: the mapper

A quick recap from 1.6. Every class has a Class object, and getDeclaredFields() hands you one Field object per field. field.get(obj) reads a field from one object, and setAccessible(true) lets you past private. getDeclaredConstructor().newInstance() builds an object, and field.set writes into it, final fields included. And an annotation kept at RUNTIME is there for getAnnotation to find.

Code that copies your objects into JSON and back is called an object mapper. No more with chains by hand. Ours is JsonMapper.java, and everything it offers fits in two lines.

1
2String text = JsonMapper.toJson(ada);
Person back = JsonMapper.fromJson(text, Person.class);

toJson takes any object and hands back JSON text. fromJson takes text and a Class token, 1.4's trick, and hands back an object of that class. Both are static. Like Json, it's a front door with nothing to build.

So Stage 4 comes in three goes. 4a sends objects out. 4b brings them back. 4c writes down, honestly, where it breaks.

4a — Your objects out to JSON

Here are the two classes we'll send through it, both in src/stage4.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18static int created = 0;                          // static: skipped by the mapper

private String name;
private int born;
@JsonName("lives_at")
private Address address;
private List<String> languages = new ArrayList<>();

public Person() {
    created++;
}

public Person(String name, int born, Address address) {
    this();
    this.name = name;
    this.born = born;
    this.address = address;
}

Line 1 is a static counter, like 1.6's Player.created. Every constructor adds one, so later we can count how many Persons got built.

Lines 3 to 7 are four private fields. Above address sits the one thing in this file that knows JSON exists, a tag: in JSON, call this field lives_at.

Lines 9 to 11 are a constructor with no parameters, which the mapper needs. Lines 13 to 18 are the one you call yourself. Its first line, this();, means "run my other constructor first", so the counter goes up either way.

The rest is getters, plus speaks, which adds a language. Address.java is two private fields, city and postcode, and the same two constructors. Nothing JSON about it at all. The tag is declared in a file of its own, and it's 1.6's @Secret with two small things added.

1
2
3
4
5
6
7
8
9
10
11import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface JsonName {

    String value();
}

Lines 1 to 7 you know from 1.6: kept at run time, fields only. Line 8 makes it public, because it's meant for other people's classes.

Line 10 is new. String value(); gives the tag a setting, a piece of text each use fills in, written like a method with no body. So @JsonName("lives_at") fills value with "lives_at". When value is the only setting, you may leave out value =.

Three helpers first

The mapper's reflection lives in three small helpers. Meet them first, because the blanks below are yours to fill with them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23import java.lang.reflect.Field;
import java.lang.reflect.Modifier;

private static Object read(Field field, Object owner) {
    try {
        field.setAccessible(true);
        return field.get(owner);
    } catch (IllegalAccessException e) {
        throw new JsonMappingException("cannot read " + field.getName(), e);
    }
}

private static boolean isSkipped(Field field) {
    return Modifier.isStatic(field.getModifiers()) || field.isSynthetic();
}

private static String jsonName(Field field) {
    JsonName annotation = field.getAnnotation(JsonName.class);
    if (annotation == null) {
        return field.getName();
    }
    return annotation.value();
}

Lines 1 and 2 fetch Field and Modifier, which the real file imports at the top.

read(field, owner) hands back what that field holds in that one object. It's 1.6's two lines, wrapped in the shape 1.5 taught. field.get throws a checked exception, and nobody calling toJson wants a try for it. So read throws an unchecked JsonMappingException instead, with the original as the cause. An int field comes back boxed, as an Integer.

isSkipped asks "should the JSON leave this out?" field.getModifiers() packs the words before a field into one int. Modifier.isStatic(...) asks whether static is among them. So Person.created is skipped: it belongs to the class, not to this person. It also skips fields javac wrote itself, which the box below explains.

jsonName is 1.6's getAnnotation, put to work. No tag? Use the field's own name. A tag? Use the text inside it.

And the mapper's exception class is the twin of JsonParseException.

1
2
3
4
5
6
7
8
9
10public class JsonMappingException extends RuntimeException {

    public JsonMappingException(String message) {
        super(message);
    }

    public JsonMappingException(String message, Throwable cause) {
        super(message, cause);
    }
}

Line 1 makes it unchecked, for the same reason. Lines 3 to 5 take a message. Lines 7 to 9 take a message and a cause, 1.5's super(message, cause), the one read uses.

toTree — the two blanks are yours

toJson does almost nothing itself. It's one line: return toTree(value).toJson();. toTree turns any object into a JsonValue tree, and the tree already knows how to print itself. So toTree is the method that matters, and its deciding line is yours.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31public static JsonValue toTree(Object value) {
    if (value == null) {
        return JsonNull.INSTANCE;
    }
    if (value instanceof String) {
        return new JsonString((String) value);
    }
    if (value instanceof Boolean) {
        return new JsonBoolean((Boolean) value);
    }
    if (value instanceof Number) {
        return new JsonNumber(((Number) value).doubleValue());
    }
    if (value instanceof List) {
        ArrayList<JsonValue> items = new ArrayList<>();
        for (Object item : (List<?>) value) {
            items.add(toTree(item));
        }
        return new JsonArray(items);
    }
    Class<?> type = value.getClass();
    checkPlainClass(type);
    LinkedHashMap<String, JsonValue> members = new LinkedHashMap<>();
    for (Field field : type.getDeclaredFields()) {
        if (isSkipped(field)) {
            continue;
        }
        members.put(/* BLANK 1 — the key */, /* BLANK 2 — the value */);
    }
    return new JsonObject(members);
}

It's a list of questions: what kind of thing is this? Lines 2 to 4 turn Java's null into JSON's null. Lines 5 to 10 turn text into a JsonString and true-or-false into a JsonBoolean. Boolean, capital B, is the object version of boolean, as 1.1's Integer was for int.

Lines 11 to 13. Number is the parent class of Integer, Long, Double and the rest, so one if catches them all, boxed ints included. .doubleValue() turns any of them into a double, because every JSON number here is a double. Remember that line. 4c sends it a bill.

Lines 14 to 20. A List becomes a JsonArray, with toTree called on each item. That's toTree calling itself, like the parser, with the simple values above as base cases.

Lines 21 to 23 handle everything else, and they start 1.6's dump with a new job. Get the Class object, let checkPlainClass refuse a few kinds (4c covers which), and make an empty map. Class<?> means a Class of some type we won't name: a ? on its own is 1.4's wildcard with no bound.

Lines 24 to 29 walk the fields, skipping the ones isSkipped says to. Line 30 builds the JsonObject from the map.

Line 28 is yours. It puts one pair into the map for this field.

Blank one is the key. It's the tag's name if there's a tag, and the field's own name if not. One helper call does exactly that.

Blank two is the value, and it's two calls, one inside the other. First get what this field holds in this object. The object is called value here, the method's parameter. Then turn whatever you got into a tree. Read it inside out, the way line 12's ((Number) value).doubleValue() reads.

Now watch the finished toTree run on Ada. Every field's value goes through toTree again, so each one gets a plate of its own.

4b — Back from JSON

Coming back is harder. Going out, the object tells you what everything is. Coming back, all you have is text, and a Class token saying what it should become.

fromJson(text, type) is one line: return fromTree(Json.parse(text), type);. That's Stage 3's parser, so bad text stops right there, with its line and column. Only real JSON reaches the mapper. fromTree hands the tree to convert, and casts the answer with 1.4's type.cast.

convert is the mirror of toTree: "what should this become?" Its first line is private static Object convert(JsonValue json, Type type). A Type is reflection's word for any type Java can write down. A Class object is one kind. A type with angle brackets, like List<String>, is the other, and it waits until after the next prediction. Here's the main part.

One helper first, because two blanks lean on it. expect(json, JsonString.class) is 1.7's typed getter in disguise. It checks the tree holds that kind of value with isInstance, and hands it back with cast. Anything else, and it throws: expected a JsonNumber but found "1906".

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27Class<?> c = (Class<?>) type;
if (json == JsonNull.INSTANCE) {
    if (c.isPrimitive()) {
        throw new JsonMappingException("null cannot go into a field of type " + c.getName());
    }
    return null;
}
if (c == String.class) {
    return /* BLANK 3 — check it's a JsonString, and hand back its text */;
}
if (/* BLANK 4 — is this a true-or-false field, of either kind? */) {
    return expect(json, JsonBoolean.class).value();
}
if (c == double.class || c == Double.class) {
    return expect(json, JsonNumber.class).value();
}
if (c == int.class || c == Integer.class) {
    return (int) whole(json, Integer.MIN_VALUE, Integer.MAX_VALUE, "int");
}
if (c == long.class || c == Long.class) {
    return whole(json, Long.MIN_VALUE, Long.MAX_VALUE, "long");
}
if (c == List.class) {
    throw new JsonMappingException("a List field needs its item type, like List<String>");
}
checkPlainClass(c);
return toObject(expect(json, JsonObject.class), c);

Line 1. The angle-bracket kind is dealt with by now, so type must be a Class object. Class<?> is a Class of some type we won't name.

Lines 2 to 7. JSON null. A field that holds a ticket can hold null. But c.isPrimitive() is true for int, long, double and boolean, whose boxes hold the value itself, from 1.1. There's nowhere to put "nothing", so null is refused, by name.

Lines 8 to 16 are one question per kind of field. Comparing Class objects with == works because there's exactly one per class, even int.class.

Blank three turns a JSON string into a Java String. Use expect to make sure it really is a JsonString. Then ask that JsonString for its text, the way 1.7's typed getters did.

Blank four is the question for a true-or-false field. Remember that a field can be declared with the small-letter type or with its capital-letter box. Each has its own Class object.

Lines 17 to 22. Whole numbers go through whole, which checks there's nothing after the point and the number fits. 1.5 into an int is refused, and so is 3000000000.

Lines 23 to 25 refuse a bare List, for a reason the next prediction shows. Lines 26 and 27: anything else must be a plain class, and the JSON an object. Then toObject does the real work.

And here is toObject, the method the last line of convert hands the real work to.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20private static Object toObject(JsonObject json, Class<?> type) {
    Object object = newInstance(type);
    for (Field field : type.getDeclaredFields()) {
        if (isSkipped(field)) {
            continue;
        }
        String name = jsonName(field);
        if (json.has(name) == false) {
            continue;                            // missing in the JSON: keep the default
        }
        Object value;
        try {
            value = convert(json.get(name), field.getGenericType());
        } catch (JsonMappingException e) {
            throw new JsonMappingException(type.getSimpleName() + "." + field.getName() + ": " + e.getMessage(), e);
        }
        write(field, object, value);
    }
    return object;
}

Line 2 builds a blank object with a helper, newInstance. It's 1.6's blank, plus setAccessible(true) so even a private constructor can run. A class with no such constructor gets a clear message: Person needs a constructor with no parameters. Lines 3 to 7 walk the fields, skip what toTree skips, and look up each JSON name. Lines 8 to 10: a key missing from the JSON leaves the field as the constructor set it.

Line 13 converts the value into whatever the field needs, by calling convert again. Line 17 writes it in, with 1.6's field.set, wrapped the way read was.

Lines 12 to 16 are for the error messages. If one field fails, toObject throws a new exception with the class and field glued on the front. The old one is its cause. So a mistake three objects deep reads as one path: AllTypes.point: Point.y: expected a JsonNumber but found true. That's a real one, from the tests.

How does it know what goes in the list?

Look at line 13 of that sample again. It hands convert the field's getGenericType(), not its getType(). For most fields those are the same. For languages, there's a puzzle.

1.4 showed you that generics are erased. While the program runs, a list of Strings and a list of numbers are the same class. So when JSON hands the mapper ["English","French"], how could it know they should be Strings? FieldTypes.java asks three questions about languages: the list object's class, the field's plain type, and its generic type.

$ cd stage4
$ java FieldTypes
the object:        java.util.ArrayList
the field, plain:  java.util.List
the field, full:   java.util.List<java.lang.String>

For a field like languages, getGenericType() hands back a ParameterizedType: a type with its angle brackets still attached. convert has a branch for it, first in the method. It reads String out of the brackets, then converts each item of the JSON array to that.

1
2
3
4
5
6
7
8
9
10if (type instanceof ParameterizedType) {
    ParameterizedType generic = (ParameterizedType) type;
    if (generic.getRawType() != List.class) {
        throw new JsonMappingException("unsupported type " + type);
    }
    if (json == JsonNull.INSTANCE) {
        return null;
    }
    return toList(expect(json, JsonArray.class), generic.getActualTypeArguments()[0]);
}

The round trip

JsonDemo.java in stage4 is the whole library in one run.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19Person ada = new Person("Ada Lovelace", 1815, new Address("London", null));
ada.speaks("English");
ada.speaks("French");

String text = JsonMapper.toJson(ada);
System.out.println(text);

Person back = JsonMapper.fromJson(text, Person.class);
System.out.println(back.name() + ", born " + back.born() + ", lived in "
        + back.address().city() + ", spoke " + back.languages() + ".");
System.out.println("Same object? " + (back == ada));
System.out.println("Person constructors run: " + Person.created);

String typo = "{\"name\":\"Grace Hopper\",\"born\":\"1906\"}";
try {
    JsonMapper.fromJson(typo, Person.class);
} catch (JsonMappingException e) {
    System.out.println("Refused: " + e.getMessage());
}

Lines 1 to 3 build Ada, with no postcode and two languages. Lines 5 and 6 send her out as JSON. Lines 8 to 10 read that text back as a brand new Person. Line 11 asks the 1.1 question: same object? Line 12 prints the counter. Lines 14 to 19 try JSON that's wrong in a quiet way. Look closely at "1906". It's in quotes.

$ cd stage4
$ java JsonDemo
{"name":"Ada Lovelace","born":1815,"lives_at":{"city":"London","postcode":null},"languages":["English","French"]}
Ada Lovelace, born 1815, lived in London, spoke [English, French].
Same object? false
Person constructors run: 2
Refused: Person.born: expected a JsonNumber but found "1906"

4c — Where it breaks, in writing

A library is only as trustworthy as its list of limits. Ours has six. The first five are pinned down by tests. The sixth is a crash. And Limit 1 surprises everybody.

Limit 1: big whole numbers

Social networks give every post an id, a big whole number. Here's a class holding one, chosen carefully.

1
2
3
4
5
6Tweet t = new Tweet();
t.id = 9007199254740993L;
System.out.println("in Java:   " + t.id);
System.out.println("as JSON:   " + JsonMapper.toJson(t));
Tweet back = JsonMapper.fromJson("{\"id\":9007199254740993}", Tweet.class);
System.out.println("read back: " + back.id);

Tweet is a tiny static nested class with one long field, id. Line 2 puts a sixteen-digit number in it, with 0.2's L for long. Line 3 prints it from Java. Line 4 sends it out as JSON. Lines 5 and 6 read the same digits back in from text, and print what arrived. A long holds nineteen digits. This has sixteen. Nothing should overflow.

$ cd stage4
$ java BigNumber
in Java:   9007199254740993
as JSON:   {"id":9007199254740992}
read back: 9007199254740992

$ cd stage4
$ java DoubleLadder
9007199254740990  ->  9007199254740990
9007199254740991  ->  9007199254740991
9007199254740992  ->  9007199254740992
9007199254740993  ->  9007199254740992
9007199254740994  ->  9007199254740994
9007199254740995  ->  9007199254740996
9007199254740996  ->  9007199254740996

Limits 2, 3 and 4

Limits.java runs the next three in one go.

1
2
3
4
5
6
7
8
9
10
11static class Pet {
    String name = "Rex";
}

static class Dog extends Pet {
    boolean goodBoy = true;
}

static class Scores {
    HashMap<String, Integer> byName = new HashMap<>();
}

A Pet with a name, and a Dog that extends it, from 1.3. A class holding a HashMap. main sends a Dog out. It reads {"nmae":"Bruno"} into a Pet, spelt wrong on purpose. And it tries to send out a Scores.

$ cd stage4
$ java Limits
a Dog:      {"goodBoy":true}
a typo:     name is still Rex
a HashMap:  cannot map a java.util.HashMap; use a plain class, a List, or a simple value

Limit 2: the dog lost its name. getDeclaredFields() means fields declared in this class, and name is declared in Pet. Dog.class.getSuperclass() hands back the parent's Class object, Pet.class, and asking again gives Object.class. Walking up that chain is a job for Section 5, Make it yours.

Limit 3: the typo vanished. nmae isn't a field, so the mapper ignored it. name wasn't in the JSON, so it kept its default. Each rule is sensible alone. Together they mean a misspelt key fails in silence.

Limit 4: the HashMap was refused. checkPlainClass refuses any class whose name starts with java., plus arrays. You saw why in 1.6: Java's own classes sit inside modules, and setAccessible would be refused. Better a clear message now than that trace later.

Limit 5: field order

The keys come out in getDeclaredFields() order. On this JVM that's the order you wrote them, but 1.6 quoted its documentation: "not in any particular order". A habit, not a promise. Every test relies on it, and they'd tell you if it changed.

Limit 6: an object that points at itself

$ cd stage4
$ java SelfLoop
Exception in thread "main" java.lang.StackOverflowError
… a few lines inside Java's own code, then …
	at JsonMapper.toTree(JsonMapper.java:66)
	at JsonMapper.toTree(JsonMapper.java:66)

Section 5Make it yours

Ideas, not instructions. Everything below is reachable with what you know. Pick the one that appeals. Something you chose yourself is worth five exercises somebody set you.

• Parent fields. Fix Limit 2. In toTree and toObject, walk from the class up through getSuperclass() until you reach Object.class, and visit each one's fields.
• Catch the loop. Fix Limit 6. Keep a list of the objects toTree is in the middle of, and throw a JsonMappingException if one turns up again.
• Strict mode. Fix Limit 3. Make fromJson refuse a key that matches no field, and say which key.
• Pretty printing. Write a toPrettyJson that puts each key on its own line, indented two spaces per level, like ada.json.
• Exact numbers. Make JsonNumber keep its text, and turn it into a long or a double only when asked. Then watch the LIMIT: test fail, which is good news.

Section 6Check yourself

Section 7What broke, and what surprised you

Twelve programs is plenty of room to go wrong. Before you close this page, write down — actually write — the answers to these:

• Which predictions did you get wrong? For each one, what were you picturing instead?
• Did you guess column 1 or column 2 for "langs"? What moved it?
• Draw the plates for parsing [[1]]. How many parseArray plates are there at the deepest moment?
• Why did broken.json's error point at line 4? Would you change that, now you know why?
• Did "two constructors" surprise you? Which one had you forgotten?
• Which of the six limits would bite you first in a real program?

If a run didn't match this page, trust your terminal and go and find out why. Every output here came from a real run on JDK 17. But your machine is the one that counts.

Section 8Everything you built in Phase 1 — and why Phase 2 exists

Eight chapters ago, a variable held a number, and you'd never written a class you could use twice. Look at what you've got now.

That's a real JSON library, with a parser and an object mapper, written from nothing. And it has tests. TestsByStage.java runs them one stage's worth at a time, and prints the running total after each.

$ cd stage4
$ java TestsByStage
Stage 1  the value tree     52 passed
Stage 2  typed getters      69 passed
Stage 3  parser and files   140 passed
Stage 4  the object mapper  186 passed
failed along the way: 0

52 checks on the tree. 17 more for the typed getters. 71 for the parser, its error messages and the file reading. 46 for the mapper, including the first five limits, each written down as a test so it can't surprise anyone. Then the real thing, the file you'd actually run:

$ cd stage4
$ java JsonTests
186 passed, 0 failed

One hundred and eighty-six. That number is yours.

Now the honest part

Every one of those tests passes. And still, some of the most important code in the library isn't yours. Ask the terminal. grep searches files for a piece of text, and -l makes it print only the names of the files that contain it.

$ cd stage4
$ grep -l "import java.util" Json*.java Person.java
JsonArray.java
JsonMapper.java
JsonObject.java
JsonParser.java
JsonTests.java
Person.java

Six files lean on java.util. JsonArray keeps its items in an ArrayList. JsonObject keeps its keys in a LinkedHashMap. The parser and the mapper build both, all the time. You've used them since 1.4, as a customer.

And you can't yet say how either one works. An array from 0.5 has a fixed number of slots, so how does an ArrayList keep growing when you add? Finding "born" among a thousand keys would mean checking every one, so how does a map jump straight to it? 1.2 promised you'd build HashMap and its sibling HashSet from scratch in Phase 2. A list that grows is built there too.

That's next. Phase 2 opens them up and has you build them yourself: lists, stacks, queues, hash tables, trees. Its flagship is your own small version of java.util, built so it can be swapped into the projects you've already written. json-java is the obvious first customer, and these 186 tests are already sitting there, waiting to check your work.