PDF files appear to consist of a series of objects that (mostly) look like this:

<object number> 0 obj
<<

<... various different kinds of data ...>

>>

The object numbers are 1-based. You don't need to write the objects in order, so you can write object 3 before object 1 if you want, but you do have to write an index table of offsets at the end of the file. It looks like you could pick sparse object numbers, but you'd end up with bogus lines in the index table (at a minimum) and maybe have to write actual empty object definitions, so you're probably better off starting at 1 and using every number from there on up.

Additionally, a lot of objects refer to other objects. For example, the root "Catalog" object refers to a "Pages" object, which refers to a list of "Page" objects. All these references need to use the number of the referenced object.

I'm cribbing heavily from FPDF, which has a series of different, very clever solutions for managing these object references, involving a mixture of hard-codes, guesses, and maybe a pinch o' the ol' luck.

So far, I'm writing a PDF which technically opens but isn't showing text yet. An identical-on-the-face-of-it PDF from FPDF shows text, but I can't figure out what's different with diff since the FPDF output is in a wacky order.

Hopefully this is just FPDF being wacky and not some kind of actual requirement for PDFs to work. I'm somewhat heartened and somewhat confused that Print → Save as PDF produces output in a different order, albeit also a wacky non-sequential order.

Also, this is sort of interesting -- the Mac PDF does this:

10 0 obj
<< /Length 11 0 R ...

...
>>
11 0 obj
1116
endobj

I think that instead of encoding "/Length 1116", it's encoding "Length $OBJECT_11" and then writing the length into object #11. Powerful!

Just some observations after popping open PDF invoices which have shown up in my inbox recently:

• Google's PDF has no data identifying the generator.
• MailGun's PDF is generated by wkhtmltopdf. (I did attempt to get this working, but was less excited about it when I realized it requires the host be running an X11 server to work, or requires compiling against a patched QT? This felt like a very deep swamp to wade into.)
• According to the document metadata, SendGrid's PDF is generated by "Zachary Nelson", an accountant who works at SendGrid. The "Aspose.Words" Java module is used to write a Word file, which Zachary opens in Word and prints to PDF. He then sends you an email with the PDF attached. 😃

I've worked with wkhtmltopdf several times, and I don't remember it actually requiring any special setup - in all cases, I basically dropped the binary in the machine (Although it's possible all my machines had X11 built in for some reason? I've never intentionally installed X11).

When I apt-get install wkhtmltopdf (or something like that) on aux001, which is Ubuntu 14, I get this:

$ wkhtmltopdf
...

Name:
  wkhtmltopdf 0.9.9

...

Reduced Functionality:
  This version of wkhtmltopdf has been compiled against a version of QT without
  the wkhtmltopdf patches. Therefore some features are missing, if you need
  these features please use the static version.

  Currently the list of features only supported with patch QT includes:

...

 * Running without an X11 server.

...

Then:

$ wkhtmltopdf http://example.org out.pdf
wkhtmltopdf: cannot connect to X server 

This may not actually be difficult to resolve, but I'd like to be convinced that generating PDFs is genuinely hard before we add a Webkit dependency, even if X11 isn't really a dependency. So far, I haven't hit any major issues.

Just my two cents here, but chrome has an headless mode that allows to "print to pdf" : https://developers.google.com/web/updates/2017/04/headless-chrome

But it might require an X server too

Yeah, I'm really hoping to not require us to run an entire browser or depend on an external service to generate PDFs. The approaches I found in my research were:

I'd eagerly trade away almost any other benefit to avoid adding an external dependency we don't control.

For example, if the resulting PDF looks incredibly stupid but we don't have to keep an eye on security vulnerabilities in all of Chrome, that's a huge win in my book.

Since master can already generate stupid-looking PDFs after I spent an afternoon fiddling with it, I'd be surprised if we end up on any path other than first-party PDF generation, although I haven't actually made it through the Postscript part yet.

you can add libreoffice/soffice to your list. last time we had to do sth like this we used soffice --headless --convert-to pdf
not really better than running a browser though... in our use case we needed to use merchant uploaded doc/docx templates in combination with our data. so even less options.
but yeah most options around pdf generation are clunky.