Uniform page numbers in PDF & EPUB

[phoneydev]

I'm new at ebook creation and am trying to sort out the page numbering
issue, specifically as it regards to nonfiction works where consistent
page numbers (regardless of font size, margins, etc. in the reader
software) tend to be more important. That is, the "normal" way of
doing things appears to be specifying page breaks in the ebook such
that the same page number in the ebook is the same page number in the
PDF which is the same page number in the printed book (presumably
printed from the PDF).

It appears, looking at the contents of an EPUB I have lying around,
that page numbers are specified by an anchor with an id (e.g. <a
id="page34"/>) which can be anywhere in the content, including the
middle of a word (e.g. pro<a id="page204"/>vides). Is this the
standard way of marking them? Is there (or could there be) any
function in Pandoc that sets the EPUB page numbers in such a way as to
synchronize with a PDF? If not in Pandoc, is it possible to post-
process the EPUB and achieve the same result?

Thanks!

[John MacFarlane]

There are two possibilities here:

(1) specify the page breaks in the markdown source; they would then
turn into page break commands in latex and the id tags in EPUB.

(2) create a PDF, then scan it to figure out where the page breaks
are, and insert tags accordingly in EPUB.

(1) would be pretty easy, but ugly, and it would not produce
nice-looking results. (Also, it would be up to the user to make
sure that there is not too much content to fit on one page.)

So, (2) is presumably what you have in mind. I don't see pandoc
supporting this, but perhaps one could build a tool to postprocess.
It would be tricky -- you'd have to parse the PDF, find page breaks,
then try to match text with text in the EPUB's html source -- where
the match would not be exact. Sounds very hard!

John

Post by phoneydev
I'm new at ebook creation and am trying to sort out the page numbering
issue, specifically as it regards to nonfiction works where consistent
page numbers (regardless of font size, margins, etc. in the reader
software) tend to be more important. That is, the "normal" way of
doing things appears to be specifying page breaks in the ebook such
that the same page number in the ebook is the same page number in the
PDF which is the same page number in the printed book (presumably
printed from the PDF).
It appears, looking at the contents of an EPUB I have lying around,
that page numbers are specified by an anchor with an id (e.g. <a
id="page34"/>) which can be anywhere in the content, including the
middle of a word (e.g. pro<a id="page204"/>vides). Is this the
standard way of marking them? Is there (or could there be) any
function in Pandoc that sets the EPUB page numbers in such a way as to
synchronize with a PDF? If not in Pandoc, is it possible to post-
process the EPUB and achieve the same result?
Thanks!
--
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
For more options, visit this group at http://groups.google.com/group/pandoc-discuss?hl=en.

[HansBKK]

Is there (or could there be) any function in Pandoc that sets the EPUB

page numbers in such a way as to synchronize with a PDF? If not in Pandoc,
is it possible to post-process the EPUB and achieve the same result?

(1) specify the page breaks in the markdown source; they would then turn
into page break commands in latex and the id tags in EPUB.

(1) would be pretty easy, but ugly, and it would not produce nice-looking

results. (Also, it would be up to the user to make sure that there is not
too much content to fit on one page.)

Do you mean via inline LaTeX? I don't see any page-break feature in
markdown.

I suppose another alternative would be to "print" the PDF from a decent
desktop EPUB reader, but of course then you'd lose all the pretty-print
capabilities.

Too bad there isn't a more elegant solution for this, a pandoc-extension
that would only appear in the relevant outputs or something, even if it
required the user to paginate manually. . .

PS to the OP if you optimize your PDF for a given screen size, then maybe
you don't even need the EPUB, most devices will handle PDFs just
fine. mobileread.com is a good site for these issues generally, here's a
sample post <http://www.mobileread.com/forums/showthread.php?t=160755>

[r***@public.gmane.org]

Post by phoneydev
I'm new at ebook creation and am trying to sort out the page numbering
issue, specifically as it regards to nonfiction works where consistent
page numbers (regardless of font size, margins, etc. in the reader
software) tend to be more important. That is, the "normal" way of
doing things appears to be specifying page breaks in the ebook such
that the same page number in the ebook is the same page number in the
PDF which is the same page number in the printed book (presumably
printed from the PDF).
It appears, looking at the contents of an EPUB I have lying around,
that page numbers are specified by an anchor with an id (e.g. <a
id="page34"/>) which can be anywhere in the content, including the
middle of a word (e.g. pro<a id="page204"/>vides). Is this the
standard way of marking them? Is there (or could there be) any
function in Pandoc that sets the EPUB page numbers in such a way as to
synchronize with a PDF? If not in Pandoc, is it possible to post-
process the EPUB and achieve the same result?

There *is* a standard markup for original page numbers, however
that is in TEI XML not HTML. You would then tranaform from TEI
via XSLT to HTML. As long as EPUB is based on HTML you cannot
use TEI markup directly. Everything else is nonstandard and adhoc
so you're free to use whatever you think fits.

Regards,

ralf

[HansBKK]

Thanks for the TEI pointer, first I'd heard of that.

I realize there are many ad-hoc ways to accomplish this, I'm just wondering
why this hasn't be implemented as a pandoc extension to markdown. Aren't
page numbers required for hard copy output?

[John MacFarlane]

Post by HansBKK
Thanks for the TEI pointer, first I'd heard of that.
I realize there are many ad-hoc ways to accomplish this, I'm just
wondering why this hasn't be implemented as a pandoc extension to
markdown.

Because pandoc doesn't do page layout, which is needed to get
pagination. If you try to guess and put page break markers in
the source text, the result will likely look terrible.

[HansBKK]

Because pandoc doesn't do page layout, which is needed to get pagination.
If you try to guess and put page break markers in the source text, the
result will likely look terrible.

By "source text" do you mean pandoc-markdown? If so, then what sort of
page-break markers do you mean, embedded latex? That combination is the
only one I've been able to find examples for in a pandoc context.

Apparently this is native for reST - what will pandoc do with it?

.. raw:: pdf
PageBreak


Presumably using the native commands latex source wouldn't be an issue,
right?

And I would expect for non-paged output like regular HTML, such directives
would just get ignored?

If pandoc doesn't handle this aspect well, are there alternative transform
tools for say LaTeX to EPUB?

[John MacFarlane]

Post by John MacFarlane
Because pandoc doesn't do page layout, which is needed to get
pagination. If you try to guess and put page break markers in the
source text, the result will likely look terrible.
By "source text" do you mean pandoc-markdown? If so, then what sort of
page-break markers do you mean, embedded latex? That combination is the
only one I've been able to find examples for in a pandoc context.
Apparently this is native for reST - what will pandoc do with it?

My point is more general. No matter what format you use for source
text -- markdown, rst, latex, html -- and no matter what mechanism
you use to mark page breaks (including mechanisms we could invent)
-- you won't get good results putting page breaks manually
in the source text. The only way to know where the page breaks should go
is to actually lay out the page. And that's a job for latex, not pandoc.

As I said in my original reply, the only way I can see to insert page
number markers in EPUB that would ennsure that the pagination matches
a latex-produced PDF would be to parse the output of latex and somehow
match it with parts of the source, inserting pagebreak markers
appropriately. Maybe someone can come up with a good way to do this, but
it's not an easy task, and in any case not a task for pandoc.

John

[HansBKK]

My point is more general. No matter what format you use for source text
-- markdown, rst, latex, html -- and no matter what mechanism you use to
mark page breaks (including mechanisms we could invent) -- you won't get
good results putting page breaks manually in the source text. The only way
to know where the page breaks should go is to actually lay out the page.
And that's a job for latex, not pandoc.

OK, so in the context of page numbering my takeaway on this is to generate
latex and then edit within / convert from there for any other "paged media"
output formats - and thanks Harald for the plastex reference.

However now getting away from the arbitrary or "every-page" context, say I
have a need to make sure that a given *structure* always starts on a new
page. I see that reST for example does allow for that in its stylesheets,
not only for obvious things like chapters, but say I wanted a "summary" or
"endnotes" section at the end of each chapter to force a new page.

Is there an equivalent if I'm using pandocs-markdown? As the simplest
example, if I want to put an entire "book" in a single file but of course
have chapters start on a new page? Or is that always taken care of by the
target output's equivalent of a stylesheet?

[John MacFarlane]

Post by John MacFarlane
My point is more general. No matter what format you use for source
text -- markdown, rst, latex, html -- and no matter what mechanism
you use to mark page breaks (including mechanisms we could invent)
-- you won't get good results putting page breaks manually in the
source text. The only way to know where the page breaks should go is
to actually lay out the page. And that's a job for latex, not
pandoc.
OK, so in the context of page numbering my takeaway on this is to
generate latex and then edit within / convert from there for any other
"paged media" output formats - and thanks Harald for the plastex
reference.
However now getting away from the arbitrary or "every-page" context,
say I have a need to make sure that a given *structure* always starts
on a new page. I see that reST for example does allow for that in its
stylesheets, not only for obvious things like chapters, but say I
wanted a "summary" or "endnotes" section at the end of each chapter to
force a new page.
Is there an equivalent if I'm using pandocs-markdown? As the simplest
example, if I want to put an entire "book" in a single file but of
course have chapters start on a new page? Or is that always taken care
of by the target output's equivalent of a stylesheet?

There's no fully general way to do this. Some of the output formats don't
even have the concept of a page (e.g. HTML).

LaTeX will take care of putting chapters on new pages.

If you're targeting HTML and you want "chunked" output, with each
chapter or section in a separate HTML page with links between them,
then the easiest thing to do is generate docbook or texinfo with
pandoc, and use their tools to generate chunked HTML.

[HansBKK]

Post by John MacFarlane

Post by HansBKK
OK, so in the context of page numbering my takeaway on this is to
generate latex and then edit within / convert from there for any other
"paged media" output formats - and thanks Harald for the plastex
reference.
However now getting away from the arbitrary or "every-page" context,
say I have a need to make sure that a given *structure* always starts
on a new page. I see that reST for example does allow for that in its
stylesheets, not only for obvious things like chapters, but say I
wanted a "summary" or "endnotes" section at the end of each chapter to
force a new page.
Is there an equivalent if I'm using pandocs-markdown? As the simplest
example, if I want to put an entire "book" in a single file but of
course have chapters start on a new page? Or is that always taken care
of by the target output's equivalent of a stylesheet?

There's no fully general way to do this. Some of the output formats don't
even have the concept of a page (e.g. HTML).
Yes, I was specifically talking here about those output formats that do,

hence "paged media".

Post by John MacFarlane
LaTeX will take care of putting chapters on new pages.

As will reST/Sphinx via stylesheets, but I guess from what you're saying
that would require not using pandoc, but the native docutil transform tools
instead.

If you're targeting HTML and you want "chunked" output, with each chapter

Post by John MacFarlane
or section in a separate HTML page with links between them, then the
easiest thing to do is generate docbook or texinfo with pandoc, and use
their tools to generate chunked HTML.

I also came across HTMLdoc <http://www.htmldoc.org/>, which goes from
"flat" HTML to "chunked" as well as postscript and PDF directly. I imagine
that EPUB could also be generated from the chunked/indexed output.

I've concluded from my research so far that in the free software space the
PDF-->EPUB path via Calibre (with perhaps some native EPUB editing in
Sigil) would be the most likely toolset to handle the coordinated
page-number aspect without custom programming-level skills. Calibre even
includes options for outputting to specific mobile devices and screen
resolutions.

Apparently the new Adobe suite that includes Framemaker and Robohelp can
handle much of this in the payware arena.

Perhaps such functionality could be considered as a "wishlist" for future
pandoc development - of course to apply to "paged media" only, ignored for
those where it's irrelevant. Just for reference, here's what I would
consider appropriate target output formats - anyone thinking of others
please speak up:

LaTeX
PDF
EPUB
RTF
ODT / OpenDocumentXML (is there a functional difference when using
LibreOffice?)

[hgeyer-Rn4VEauK+AKRv+]

Hi!

Post by HansBKK
If pandoc doesn't handle this aspect well, are there alternative transform
tools for say LaTeX to EPUB?

If you don't want any2any conversion, but instead want the power of
LaTeX and still generate other output formats, then probably something
like plasTeX http://plastex.sf.net/ might be more useful to you.

plasTeX does support EPUB, but I haven't tried it yet (only plain text
and xhtml), but I'll need EPUB soon, so I'm willing to help fixing
problems ...

HTH,
Harald

[HansBKK]

I thought it best to continue this topic in the right place rather than
"amazing software". Keep in mind I'm a complete non-programmer and in fact
all-round noob in this domain, so the below may not make a lot of sense.

The first step to getting Pandocs to help with this would be to first make
sure that ConTeXt or LaTeX are capable of outputting a properly accessible
structured & tagged PDF. The next step is to determine how to get that TeX
code output from reST and/or p-markdown input.

The PDF could be converted to EPUB using something like Calibre, and then
reverse-engineered so that the EPUB writer would be able to do the
conversion directly from that same source markup.

Here<http://media.wiley.com/product_data/excerpt/16/04706129/0470612916-4.pdf>'s
an informative sample chapter from an Acrobat book.

[HansBKK]

Post by HansBKK
The first step to getting Pandocs to help with this would be to first make
sure that ConTeXt or LaTeX are capable of outputting a properly accessible
structured & tagged PDF. The next step is to determine how to get that TeX
code output from reST and/or p-markdown input.

I must have missed the fact that pandoc now outputs PDF directly.

Although I have to believe that most people caring about such "edge-case"
issues as having their EPUB pagination match their hard copy would create
PDF via TeX (IOW LaTeX or ConTeXt), that may well not be the case.

As p-markdown stands now, they would *have* to go that route (p-markdown
--> TeX --> PDF), but it might be worth adding to the wishlist that somehow
the native syntax could support this feature when converting directly to
both PDF and EPUB.

If none of this is likely to happen anytime soon, then it seems the
possible paths to achieve this are

pandoc --> PDF --> Adobe payware? --> EPUB

pandoc --> ?? --> Calibre --> EPUB
pandoc --> ?? --> Calibre --> PDF

When I get a round tuit, I'll post my findings back here

[Dirk Laurie]

Post by HansBKK
I must have missed the fact that pandoc now outputs PDF directly.

Pandoc will create a latex file and use pdflatex (or another engine, see
--latex-engine) to convert it to PDF:

[Martin Schröder]

Post by HansBKK
The first step to getting Pandocs to help with this would be to first make
sure that ConTeXt or LaTeX are capable of outputting a properly accessible
structured & tagged PDF. The next step is to determine how to get that TeX
code output from reST and/or p-markdown input.

ConTeXt MkIV can create tagged PDF.

Best
Martin

[HansBKK]

Post by Dirk Laurie
Pandoc will create a latex file and use pdflatex (or another engine, see

--latex-engine) to convert it to PDF:

Aha, I see. Well good, then no need to reverse-engineer how ConTeXt does
it, just one path to both.

Post by Dirk Laurie

Post by HansBKK
The first step to getting Pandocs to help with this would be to first

make

Post by HansBKK
sure that ConTeXt or LaTeX are capable of outputting a properly

accessible

Post by HansBKK
structured & tagged PDF.

ConTeXt MkIV can create tagged PDF.

Excellent!

So the next questions are:

- is that tagging in the output PDF the right structures (not just
necessary but sufficient) to ensure page-mapping?

- what open-source tools will convert from such "page-mapped" PDF to EPUB?

pandoc --> ConTeXt --> PDF --> Adobe payware? --> EPUB

pandoc --> ConTeXt --> PDF --> FOSS? --> EPUB

I will take on determining if Calibre is a possibility there

--------------------------
or

- will tools available to ConTeXt allow it to publish such page-mapped
EPUBs directly

pandoc --> ConTeXt --> PDF
pandoc --> ConTeXt --> EPUB

If all paths to this goal go through ConTeXt, that probably implies that it
isn't necessary for Pandoc to be able to output the necessary markup, as it
would be added via post-processing?

[r***@public.gmane.org]

Post by HansBKK
- what open-source tools will convert from such "page-mapped" PDF to EPUB?
- will tools available to ConTeXt allow it to publish such page-mapped
EPUBs directly
pandoc --> ConTeXt --> PDF
pandoc --> ConTeXt --> EPUB

Honestly, the latter (not converting from PDF) would always
be better because of inherent problems with any PDF conversion.
As soon as you're starting with math fractions, or do some fancy
tables, calibre will fail you, and I know of no better free tool.


ralf

[HansBKK]

Post by HansBKK
pandoc --> ConTeXt --> PDF
pandoc --> ConTeXt --> EPUB

Honestly, the latter (not converting from PDF) would always be better
because of inherent problems with any PDF conversion.

That may be true in general, but the specific issue being discussed here
requires that this question

- will tools available to ConTeXt allow it to publish such

page-mapped EPUBs directly

be answered in the affirmative before that will be possible.

[r***@public.gmane.org]

Post by HansBKK

Post by HansBKK
pandoc --> ConTeXt --> PDF
pandoc --> ConTeXt --> EPUB

Honestly, the latter (not converting from PDF) would always be better
because of inherent problems with any PDF conversion.

That may be true in general, but the specific issue being discussed here
requires that this question

- will tools available to ConTeXt allow it to publish such

page-mapped EPUBs directly
be answered in the affirmative before that will be possible.

I was aware of that. What I meant was, I'd rather put time
in helping to implement such ConTeXt functionality than even
hoping to achieve the same by analyzing PDFs.

['Jonathan Scholbach' via pandoc-discuss]

You might have a look at
https://tex.stackexchange.com/questions/230923/get-the-automatically-produced-line-and-page-breaks-from-latex

Post by phoneydev
I'm new at ebook creation and am trying to sort out the page numbering
issue, specifically as it regards to nonfiction works where consistent
page numbers (regardless of font size, margins, etc. in the reader
software) tend to be more important. That is, the "normal" way of
doing things appears to be specifying page breaks in the ebook such
that the same page number in the ebook is the same page number in the
PDF which is the same page number in the printed book (presumably
printed from the PDF).
It appears, looking at the contents of an EPUB I have lying around,
that page numbers are specified by an anchor with an id (e.g. <a
id="page34"/>) which can be anywhere in the content, including the
middle of a word (e.g. pro<a id="page204"/>vides). Is this the
standard way of marking them? Is there (or could there be) any
function in Pandoc that sets the EPUB page numbers in such a way as to
synchronize with a PDF? If not in Pandoc, is it possible to post-
process the EPUB and achieve the same result?
Thanks!