netsgiro: a parser and builder for AvtaleGiro and OCR Giro files

Today I released the first stable release of netsgiro to PyPI. netsgiro is a Python 3.4+ library for parsing and building Nets “OCR” files.

AvtaleGiro is a direct debit solution that is in widespread use in Norway, with more than 15 000 companies offering it to their customers. OCR Giro is used by Nets and Norwegian banks to update payees on recent deposits to their bank accounts. In combination, AvtaleGiro and OCR Giro allows for a high level of automation of invoicing and payment processing.

The “OCR” file format and file format name originates in days when giro payments were delivered on paper to your bank and then processed either manually or using optical character recognition, OCR. I’m not sure how old the format is, but some of the examples in the OCR Giro specification use dates in 1993, and the specification changelog starts in 1999 and ends in 2003. A couple of decades later, the file format is still in daily use by Nets’ AvtaleGiro and OCR Giro services. In other words, I have high hopes that this will be a very stable open source project requiring minimal maintenance efforts.

Here’s an example OCR file, to be used in later examples:

>>> data = '''
... NY000010555555551000081000080800000000000000000000000000000000000000000000000000
... NY210020000000000400008688888888888000000000000000000000000000000000000000000000
... NY2121300000001170604           00000000000000100          008000011688373000000
... NY2121310000001NAVN                                                        00000
... NY212149000000140011 Gjelder Faktura: 168837  Dato: 19/03/0400000000000000000000
... NY212149000000140012                  ForfallsDato: 17/06/0400000000000000000000
... NY2121300000002170604           00000000000000100          008000021688389000000
... NY2121310000002NAVN                                                        00000
... NY212149000000240011 Gjelder Faktura: 168838  Dato: 19/03/0400000000000000000000
... NY212149000000240012                  ForfallsDato: 17/06/0400000000000000000000
... NY2121300000003170604           00000000000000100          008000031688395000000
... NY2121310000003NAVN                                                        00000
... NY2121300000004170604           00000000000000100          008000041688401000000
... NY2121310000004NAVN                                                        00000
... NY2121300000005170604           00000000000000100          008000051688416000000
... NY2121310000005NAVN                                                        00000
... NY212149000000540011 Gjelder Faktura: 168841  Dato: 19/03/0400000000000000000000
... NY212149000000540012                  ForfallsDato: 17/06/0400000000000000000000
... NY2102300000006170604           00000000000000100          008000061688422000000
... NY2102310000006NAVN                                                        00000
... NY210088000000060000002000000000000000600170604170604000000000000000000000000000
... NY000089000000060000002200000000000000600170604000000000000000000000000000000000
... '''.strip()

netsgiro is obviously not a child of recent changes in my spare time interests, but was conceived as a part of our ongoing efforts in automating all aspects of invoicing and payment processing for Otovo’s residential solar and electricity customers. As such, netsgiro is Otovo’s first open source project. Hopefully, it will soon get some company on our public GitHub profile.

As of netsgiro 1.0.0, sloccount(1) counts 1160 lines of code and 948 lines of tests, providing 97% statement coverage. In other words, netsgiro is a small library trying to do one thing well. Meanwhile, enough effort has been poured into it that I’m happy I was immediately allowed to open source the library, hopefully saving others and my future self from doing the same work again.

The library is cleanly split in two layers. The lower level is called the “records API” and is imported from netsgiro.records. The records API parses OCR files line by line and returns one record object for each line it has parsed. This is done with good help from Python’s multiline regexps and enum.IntEnum, and Hynek Schlawack’s excellent attrs. Conversely, you can also create a bunch of record objects and convert them to an OCR file.

>>> import netsgiro.records
>>> records = netsgiro.records.parse(data)
>>> len(records)
22
>>> from pprint import pprint
>>> pprint(records)
[TransmissionStart(service_code=<ServiceCode.NONE: 0>, transmission_number='1000081', data_transmitter='55555555', data_recipient='00008080'),
 AssignmentStart(service_code=<ServiceCode.AVTALEGIRO: 21>, assignment_type=<AssignmentType.TRANSACTIONS: 0>, assignment_number='4000086', assignment_account='88888888888', agreement_id='000000000'),
 TransactionAmountItem1(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, nets_date=datetime.date(2004, 6, 17), amount=100, kid='008000011688373', centre_id=None, day_code=None, partial_settlement_number=None, partial_settlement_serial_number=None, sign=None),
 TransactionAmountItem2(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, reference=None, form_number=None, bank_date=None, debit_account=None, _filler=None, payer_name='NAVN'),
 TransactionSpecification(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, line_number=1, column_number=1, text=' Gjelder Faktura: 168837  Dato: 19/03/04'),
 TransactionSpecification(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, line_number=1, column_number=2, text='                  ForfallsDato: 17/06/04'),
 ...
 AssignmentEnd(service_code=<ServiceCode.AVTALEGIRO: 21>, assignment_type=<AssignmentType.TRANSACTIONS: 0>, num_transactions=6, num_records=20, total_amount=600, nets_date_1=datetime.date(2004, 6, 17), nets_date_2=datetime.date(2004, 6, 17), nets_date_3=None),
 TransmissionEnd(service_code=<ServiceCode.NONE: 0>, num_transactions=6, num_records=22, total_amount=600, nets_date=datetime.date(2004, 6, 17))]

The higher level “objects API” is imported from netsgiro. It combines multiple records into higher level objects. For example, an AvtaleGiro payment request can consist of up to 86 records, which in the higher level API is represented by a single PaymentRequest object.

>>> import netsgiro
>>> transmission = netsgiro.parse(data)
>>> transmission
Transmission(number='1000081', data_transmitter='55555555', data_recipient='00008080', date=datetime.date(2004, 6, 17))
>>> len(transmission.assignments)
1
>>> transmission.assignments[0]
Assignment(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<AssignmentType.TRANSACTIONS: 0>, number='4000086', account='88888888888', agreement_id='000000000', date=None)
>>> len(transmission.assignments[0].transactions)
6
>>> from pprint import pprint
>>> pprint(transmission.assignments[0].transactions)
[PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=1, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000011688373', reference=None, text=' Gjelder Faktura: 168837  Dato: 19/03/04                  ForfallsDato: 17/06/04\n', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=2, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000021688389', reference=None, text=' Gjelder Faktura: 168838  Dato: 19/03/04                  ForfallsDato: 17/06/04\n', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=3, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000031688395', reference=None, text='', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=4, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000041688401', reference=None, text='', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=5, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000051688416', reference=None, text=' Gjelder Faktura: 168841  Dato: 19/03/04                  ForfallsDato: 17/06/04\n', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.AVTALEGIRO_WITH_PAYEE_NOTIFICATION: 2>, number=6, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000061688422', reference=None, text='', payer_name='NAVN')]

The higher level API also features some helper methods to quickly build payment requests, the only file variant typically created by anyone else than Nets.

>>> from datetime import date
>>> from decimal import Decimal
>>> import netsgiro
>>> transmission = netsgiro.Transmission(
... 	number='1703231',
...	data_transmitter='01234567',
...	data_recipient=netsgiro.NETS_ID)
>>> assignment = transmission.add_assignment(
... 	service_code=netsgiro.ServiceCode.AVTALEGIRO,
...	assignment_type=netsgiro.AssignmentType.TRANSACTIONS,
...	number='0323001',
...	account='99998877777')
>>> payment_request = assignment.add_payment_request(
...     kid='000133700501645',
...     due_date=date(2017, 4, 6),
...     amount=Decimal('5244.63'),
...     reference='ACME invoice #50164',
...     payer_name='Wonderland',
...     bank_notification=None)
>>> transmission.get_num_transactions()
1
>>> transmission.get_total_amount()
Decimal('5244.63')
>>> data = transmission.to_ocr()
>>> print(data)
NY000010012345671703231000080800000000000000000000000000000000000000000000000000
NY210020000000000032300199998877777000000000000000000000000000000000000000000000
NY2102300000001060417           00000000000524463          000133700501645000000
NY2102310000001Wonderland                         ACME invoice #50164      00000
NY210088000000010000000400000000000524463060417060417000000000000000000000000000
NY000089000000010000000600000000000524463060417000000000000000000000000000000000

Otovo has been using netsgiro in production for about a month now, and so far so good. We’re surely not the only shop in Norway doing invoicing with Python, so I hope netsgiro will be a useful building block for others as well. If you’re interested in learning more, start with the quickstart guide and then continue with the API reference.

Happy invoicing!

March and April contributions

The following is a short summary of my open source work in March and April, almost like in previous months, except that I haven’t spent as much time as previously on Open Source the last two months.

Debian

Mopidy

  • Bugfixes for the upcoming Mopidy 2.0.1 (which should have been released a long time ago): merged PR #1455, created PR #1493.

  • Started on, but didn’t finish, fixing the Travis CI setup for Mopidy-GMusic.

  • Upgraded the Mopidy project server from Ubuntu 14.04 LTS to 16.04 LTS. Rebuilt the discuss.mopidy.com Discourse/Docker instance.

  • Accepted Lars Kruse as the new maintainer of Mopidy-Beets. Thanks!

  • The extensions still in need of a new maintainer are:

    If you’re a user of any of these and want to contribute, please step up. Instructions can be found in the README of any of these projects.

February contributions

The following is a short summary of my open source work in February, just like in previous months.

Debian

Mopidy

  • Released Mopidy-Spotify 2.3.1: Works around search being broken in libspotify by using the Spotify Web API for the search functionality, and nothing else.

  • Wrapped up and released Mopidy 2.0 release. This was a quite large release that ports to GStreamer 1.x and adds support for gapless playback. It was uploaded to Debian in time to be part of Ubuntu 16.04 LTS.

  • Released Mopidy-Spotify 3.0.0: A couple of small compatibility fixes to work with Mopidy 2.x and GStreamer 1.x.

  • Updates to packages not in Debian, only at apt.mopidy.com:

    • Uploaded mopidy-spotify-tunigo 1.0.0-0mopidy1. Far down the chain, it depends on libspotify, which is not in Debian.
    • Uploaded mopidy-spotify 2.0.0-0mopidy1.
  • Updated homebrew-mopidy with lots of new releases:

    • Mopidy 2.0.0
    • Mopidy-Dirble 1.3.0
    • Mopidy-SoundCloud 2.0.2
    • Mopidy-Spotify 2.3.0, 2.3.1 and 3.0.0
    • python-backports-abc 0.4
    • python-backports-ssl-match-hostname 3.5.0.1
    • python-certifi 2015.11.20.1
    • python-cffi 1.5.0 and 1.5.2
    • python-requests 2.9.1
    • python-singledispatch 3.4.0.3
    • python-six 1.10.0
    • python-tornado 4.3
  • Rebased my feature/py3-compat branch on top of Mopidy 2.0.

  • Planned breaking changes for Mopidy 3.0: unbundling of Mopidy.js, removal of deprecated core APIs, model APIs, audio APIs and removal of the http/static_dir config.

  • Planned breaking changes to modernize Mopidy.js: change default calling convention to by-position-or-by-name, replace When.js with ES6 Promise, replace Bane.js with EventEmitter, and replace Buster.js with Karma+Mocha.

Comics

  • Started working on the upgrade to Django 1.8. Most deps are upgraded. The major remaining pieces is to upgrade django-registration from 0.8 to 2.0 and to replace the years-old vendored copy of django-invitation (singular) with django-invitations (plural).

More posts