ebp_docs

Latest update: 26th June 2024

Table of Contents

Overview

The API v2 is already on-line EBP API. ICO team has developed it in Python using a framework called Django. You can start testing with your own data.

We have decided the JSON structure for the data provision EBP API, as you can see in the json schema. Data provisions should inclide always to a particular date range (i.e. 2016-01-04 / 2016-01-10) and partner_data_source. For example, if ICO wants to submit Bird Garden Survey and Ornitho.cat data it should sent data separately (since they originate from two different sources).

We have also implemented three provision modes:

Events and records list follow the New EBP data standard format and the database structure proposal by BTO. JSON schema is also described in the document.

EBP data standard

The EBP standard handles three types of data: casual records, complete lists and fixed lists. Data will be provided in two main tables, events and records.

Casual records will be aggregated, by 10x10 km square and date, not by week as before.

On the other hand, the data coming from complete and fixed lists will be collected as individual records (i.e. not aggregated; one species in the lists = one record) using an event identifier to assign each record to each observational event (a list recorded in a given date and location).

Regarding location, partners could choose between giving the exact original location or the 10x10 square reference. Note that unlike with casual records, complete/fixed lists whose location is given as 10x10 squares are not aggregated (i.e. no precise location is given but the coarser location reference is not used to aggregate data).

The purpouse of Events table is to provide the contextual data (like the total number of records for all species, number of observers, etcs).

EBP workflow

Introduction

Data provision will be done at partner_source level. All events and records must be from the same partner_source. We have defined a l_partners table and l_partner_sources where you will find the partner_source_code required for the data provison.

In agreement with the EBCC decision regarding species standard taxonomy checklists, we have used the HBW species_list and codes. Through the API you can have a list of these species and also to protocols, project_types for protocols and breeding_codes.

Data flow schema

  1. System/partner sends periodically a data provision to the API
  2. The API validates the format and checks consistency inside de provision (unique event ids, dates outside range, location format, etc)
  3. The API gives a result to the system/partner with an audit id where the system can check validation errors
  4. The API continues doing some background checkings to the data (location inside partner’s area, unique event_id inside partner’s data, etc)
  5. System/partner can access to the provision errors through the API.

EBP workflow

Authentication

The EBP repository uses the OAuth 2.0 protocol for authentication and authorization with PasswordGrant credentials.

a) Request a user. A user will be created for your provided username (user mail is desirable).

b) Once the user is created, you can obtain the OAuth 2.0 credentials (client_id and client_secret) from the EBP repository Console.

c) Your portal will get the token accessing via POST /oauth/token/:

d) Use the access token to access the secured routes from the API.

curl -X GET https://api-v2.eurobirdportal.org/audit/<id>/ -H 'Authorization: Bearer ACCESS_TOKEN'

Connection scenarios

We will have three diferent connection scenarios:

Mode All data replaced Delete events Old events (outside provided date range)
Bulk mode :heavy_check_mark: :x: :x:
Standard mode :x: :heavy_check_mark: :heavy_check_mark: (updates, removals)
Test mode :x: :x: :heavy_check_mark:

Bulk mode (mode==B)

A bulk provision will be used when providing old data.

Standard mode (mode==S)

An standard provision will be used during weekly or monthly updates.

Test mode (mode==T)

The data provision will not be uploaded to the database. It will be use for testing porpouses.

Modification scenarios

During standard updates, portals should send old modified and removed data. Events and records will have a state field to differentiate updates/new from deletes.

Event modifications

When an event is modified, it should be sent again with new values and state=1. All event values will be replaced.

Type Changes Action
Event Update/insert Send the complete event again (and modifications in case of event updates) with field state = 1
Event Delete Send the event again with field state = 0

Record modifications

In case of records modifications, the event should be sent anyway. In most cases event should be also modified. We will have two different modes to provide the records.

a) Only modified records (default update mode)

Send the complete event with the list modified records only.

Type Changes Action
Record Insert Send new records with state=1
Record Update Send the updated records with state=1 again. They will be overwritten
Record Delete Send the removed records with field state=0

b) All records

Send the complete event with the full list of records.

To allow this mode record_updates_mode = A field in the data provision should be added.

Type Changes Action
Record Update/insert Send the complete list of records it doesn’t matter if they are new or modified. All records will be overwritten with the new list of records
Record Delete No need to send the removed records

Data provision structure

To send data to the EBP repository you should send a JSON structure with the described mandatory fields. This is the API method to send the data

Property Type Description
partner_source string The unique of the EBP partner source
start_date date Start date provided
end_date date Last date provided
events array[Event] List of events
records array[Record] List of records
mode string ennun[B,S,T] Provision mode B (Bulk: all data is replaced) / S (Standard: new data is provided/ T (Test: data validation porpouses)
record_updates_mode (optional) string ennun[M,A] Update mode M (Only updated records will be provided) / Update mode A (All records are provided). Default mode is M when record_updates_mode is not provided

Data provision sample

The data provision structure


{
    "mode": "S",
    "partner_source": "CAT_ORN",
    "start_date": "2016-01-04",
    "end_date": "2016-01-10",
    "events": [
        {
            "data_type": "L",
            "date": "2016-01-04",
            "event_id": "71456",
            "location": "POINT(3.056 41.813)",
            "location_mode": "E",
            "observer": "7840",
            "protocol_id": "",
            "records": 27,
            "time": "17:14:00",
            "state": 1

        }
    ],
    "records": [
        {
            "count": 2,
            "event_id": "71456",
            "flying_over": "N",
            "record_id": "3170459",
            "records_of_species": 1,
            "species_code": 52834,
            "state": 1
        },
        {
            "count": 2,
            "event_id": "71456",
            "flying_over": "N",
            "record_id": "3170448",
            "records_of_species": 1,
            "species_code": 58515,
            "state": 1
        }
    ]

}

Events and records

Within the JSON definition for records and events we won’t use nulls as the provided standard. When a field is null we can provided and empty string or remove the field from the json.

Events

The purpouse of Events table is to provide the contextual data (like the total number of records for all species, number of observers, etcs).

{
    'properties': {
        'data_type': {'enum': [ 'C', 'L', 'F' ],'description': 'C (casual record) / L (complete list) / F (fixed list /partial list)' },
        'date':  { 'type': 'string' },
        'duration': { 'type': 'number' ,'description':'Duration (in hours). Null if unknown or location_mode=A','required':False},
        'event_id':  { 'type': 'string' },
        'location_mode': {'enum': [ 'E', 'D', 'A' ],'description': 'E (original exact location provided) / D (location lowered to 10x10km level ETRS89-LAEA grid) / A (data aggregated at 10x10km level ETRS89-LAEA grid)' },
        'location': { 'type': 'string', 'description':'Centroid of the location in Well Known Text (WKT) in WGS84' },
        'observer': { 'type': 'string' },
        'protocol_id': { 'type': 'string' },
        'radius': { 'type': 'number','description':'Maximum distance (in m) to the location centroid travelled/covered during the observational event','required':False},
        'records': { 'type': 'integer', 'minimum': 1, 'description':'Total number of records.'},
        'time':  { 'type': 'string','required':False},
        'state':  { 'type': 'number'}
    },
    'type': 'object'
}

Event removal

In case of event removal, you can send again the full event with state=0 or only event_id and state field.


  {
      "event_id": "71456",
      "state": 0

  }

Records

In the new API v2 version will send records from all species.

{
    'properties': {
        'breeding_code': { 'type': 'integer', 'description':'Maximum breeding code. Codes to be provided (based on EBBA2 standard).','required':False},
        'count': { 'type': 'integer', 'minimum': 0, 'description':'Number of individuals counted (loc: E/D) or Maximum count of all records with counts. (loc: A)'},
        'event_id':  { 'type': 'string', 'description':'Identifier of the observational event (e.g. a given complete list).'},
        'flying_over':  { 'type': 'string' },
        'record_id': { 'type': 'string' , 'description': 'Identifier of the record'},
        'records_of_species':  { 'type': 'integer', 'minimum': 1, 'description':'Number of records of the given species',
        'state': {'type':'number'}
      }
    }
}

Record removal

In case of record removal, you can send again the full event with state=0 or only event_id, record_id and state field.


  {
      "record_id": "340000",
      "event_id": "71456",
      "state": 0

  }

Validation phases

We have splitted validation process in several phases. Phases 1 and 2 are done before giving the answer to the client. Phase 3 validation require more time and are sent to a queue and processed later.

  1. Global and schema validations. It checks that the data provision fits the JSON schema. Also checks that partner_source and date_range are correct.
  2. Pre-validation. It checks simple errors that can’t be checked with the schema. For example: protocol_codes, event_code_repetitions inside the same provision or species_codes repetitions inside the same event,…
  3. Post-validations. Those validations are done directly into the database. For example: points inside the partner area, correct species_codes, etc…

We’ve created the audit tables for logging the validations during data provision process. You can check through the API the list of errors for each data provision. The reply from the server will give you the audit_id to access lately to the error list.

Schema validations

JSON schema validations Error code Implemented Fields
Field should be an integer integer_format :heavy_check_mark: records
Field should be a number number_format :heavy_check_mark: duration, location_x, location_y, radius
Field should be a string string_format :heavy_check_mark: event_id, flying_over, record_id, event_id, observer, protocol_id, partner_source, mode
Required fields required_field :heavy_check_mark: partner_source, start_date, end_date, mode, events, records, data_type, count, event_id, species_code, record_id, records_of_species, data_type,date, event_id, location_mode, location_x, location_y, location, observer, protocol_id, records
Field should be a date in ISO 8601 format (YYYY-MM-DD) date_format :heavy_check_mark: start_date, end_date, date
Field should be a time in format HH:MM:SS time_format :heavy_check_mark: time
Location_mode should be E (original exact location provided), D (location lowered to 10x10km level ETRS89-LAEA grid) or A (data aggregated at 10x10km level ETRS89-LAEA grid) location_mode_format :heavy_check_mark: location
Mode should be B (bulk mode), S (standard mode) or T (test mode) mode_format :heavy_check_mark: mode

Global checks

Global cheks Error code Implemented
Partner id exists partner_not_found :heavy_check_mark:
Start date later than initial EBP date partner_not_found :heavy_check_mark:
End date not in the future old_init_date :heavy_check_mark:

Pre-checks

Pre-checks Error code Implemented
Protocol code not found protocol_not_found :heavy_check_mark:
When location_mode is A (aggregated), field value has to be null (time,duration,radius,flyingover) field_not_null_aggregated :heavy_check_mark:
Location_mode E/D records_of_species > 1 records_not_agg_gt_1 :heavy_check_mark:
Location_mode A observer different observers observer_not_number :heavy_check_mark:
Event_date outside provided range in bulk outside_date_range :heavy_check_mark:
Provided extenal_event_id is not unique, has been already provided event_id_not_unique :heavy_check_mark:
Provided extenal_record_id is not unique, has been already provided record_id_not_unique :heavy_check_mark:
Provided species_code is repeated in the same event species_code_not_unique :heavy_check_mark:
Duration should be smaller than 24 hours duration_gt_24h :heavy_check_mark:
Records must be greater than 0 zero_records :heavy_check_mark:

Post-check

Post-check Error code Implemented
Location is outside partners area outside_location :heavy_check_mark:
Species_code not found in the EBP species list species_code_not_found :heavy_check_mark:
Provided extenal_event_id in record not found in provided events event_id_not_found :heavy_check_mark:
Provided species_code when protocol data is outside fixed list    

Species list extension

The list of EBP target species has been extended from the current 137 species to all species. Species codes have been updated following the HBW-BirdLife Checklist v8.1 2024. Each system/partner will have to map the new species with its own thesaurus. You can downloading the species list in CSV format:

Old data

If you’ve already sent data to the EBP v1 repository you will send again all data with new species codes:

New data and updates

Daily updates (in standard mode) will also have to include the new species.

Examples

A) Standard mode with complete event


{
    "mode": "S",
    "partner_source": "CAT_TEST",
    "start_date": "2019-07-31",
    "end_date": "2019-07-31",
    "events": [
        {
            "data_type": "C",
            "date": "2019-07-31",
            "event_id": "10kmE374N210_1564531200",
            "location": "10kmE374N210",
            "location_mode": "A",
            "observer": "2",
            "protocol_id": "",
            "records": 104,
            "time": ""
        }
      ],
      "records": [
          {
              "count": 14,
              "event_id": "10kmE352N210_1564531200",
              "flying_over": "N",
              "record_id": "10kmE352N210_1564531200_52477",
              "records_of_species": 1,
              "species_code": 52477
          },
          {
              "count": 6,
              "event_id": "10kmE352N210_1564531200",
              "flying_over": "N",
              "record_id": "10kmE352N210_1564531200_52875",
              "records_of_species": 2,
              "species_code": 52875
          },
          {
              "count": 0,
              "event_id": "10kmE352N210_1564531200",
              "flying_over": "N",
              "record_id": "10kmE352N210_1564531200_60753",
              "records_of_species": 1,
              "species_code": 60753
          }
        ]
    }

B) Standard mode with state=2


{
    "mode": "S",
    "partner_source": "CAT_TEST",
    "start_date": "2019-07-31",
    "end_date": "2019-07-31",
    "events": [
          {
          "event_id": "10kmE352N210_1564531200",
          "state": 2
          }
      ],
      "records": [
          {
              "count": 14,
              "event_id": "10kmE352N210_1564531200",
              "flying_over": "N",
              "record_id": "10kmE352N210_1564531200_52477",
              "records_of_species": 1,
              "species_code": 52477
          },
          {
              "count": 6,
              "event_id": "10kmE352N210_1564531200",
              "flying_over": "N",
              "record_id": "10kmE352N210_1564531200_52875",
              "records_of_species": 2,
              "species_code": 52875
          },
          {
              "count": 0,
              "event_id": "10kmE352N210_1564531200",
              "flying_over": "N",
              "record_id": "10kmE352N210_1564531200_60753",
              "records_of_species": 1,
              "species_code": 60753
          }
        ]
    }

Protocols

Standard bird monitoring data is already collected by some local online portals and, therefore, we needed a standard that could handle it correctly. Properly storing this information will certainly increase the overall quality of the EBP data but also opens new possibilities in terms of data analysis and regarding the development of further synergies with other EBCC initiatives.

Note that to be able to deal with data coming from fixed lists and standard monitoring projects in general, we needed to add a third table to the ones already existing in the former standard: the tables events and records. This third table, named protocols, will collect the details of the protocol followed (e.g. a given Common Breeding Bird Survey) and, in case of fixed lists, the definition of the list.

We’ve created several API methods to define and create your own protocols into the EBP respository database. You can see the JSON protocol definition and the fields description. Once the protocol is created, you can use your created protocols code in the protocol_id field in the data provision events.


{
    "protocol_code": "ODJ",
    "title": "Ocells dels Jardins",
    "project_type": "GS",
    "method": "T",
    "website": "http://ocellsdelsjardins.cat/",
    "description": "Ocells dels Jardins is a citizen science project aimed to monitor the use of gardens and small parks by birds and other wildlife.",
    "protocol_details": "Very simple protocol. Only birds detected in the defined sampling area (i.e. garden) and that are no flying-over should be reported and, if possible, counted (maximum number detected in a given moment). A predefined list of 52 bird species and 5 non-bird species is given but poeple can report any bird species if they wish. Participants can count birds any time of the year and as many times as they wish, but it is encouraged to do so at least twice a year (last week of January and last week of May).",
    "ebp_data_structure": "Identital to original database",
    "citation": "2015. Ocells dels Jardins, Catalan Ornithological Institute",
    "id_gbif" : "",
    "geographic_coverage": "Catalonia, Spain",
    "start_year": 2014,
    "end_year": "",
    "ongoing": true,
    "link": "",
    "fixed_list_tags": "ESP(54105;54154;57821;58496;58952;58861;60925;61286;61290;53077;54565;55328;55871;57729;57746;58455;58252;60727;61340;61408);NFO"
}

Protocol fields description

Link, id_gbif, end_year and website are optional. If they are emtpy or null, it’s not necessary to send the fields in the JSON file.

project_type Project title
CB Common breeding bird survey
CW Common winter bird survey
WW Winter waterbird count
BA Breeding bird atlas
MC Migration count
WA Winter bird atlas
GS Garden bird survey
RB Rare breeding bird survey
OT other monitoring project
BR Bird ringing/banding results
NF Nocturnal flight calls survey
CL Checklist (list) project
MS Mortality survey (dead, sick, injured birds)
method Method description
P point counts
M mapping methods
L line-transect
T flexible surveys in which only time is controlled and there is no special requirement regarding the area/distance covered or speed
A area survey
O other methods
Species tags Species tag description
NFO no fly-overs
ORB only ringed/trapped birds
OBB only breeding birds
OWB only waterbirds
OSB only seabirds
ORA only raptors
ORS only raptors and soaring birds
OAM only active migrants
PLN partial list/fixed list no strict: other species can be reported
OCS only dead/injured/sick birds

API protocol methods (not implemented in v2 - use internal area)

Endpoint Method Description
/protocols GET Get list of own protocols
/protocols POST Create new protocol from the provided JSON
/protocols/{procotol_code} GET Get a concrete protocol with {protocol_code}
/protocols/{procotol_code} PUT Modify the protocol {protocol_code} with the provided JSON
/protocols/{procotol_code} DELETE Delete the protocol {protocol_code} if it’s no related events

Recomended system integration steps

1. DB/system preparation work

2. Basic data provision testing

3. Standard data flow integration

4. Complete data flow with old data

Metadata

Species list

Breeding codes

Breeding code Description
0 Non breeding (species observed but suspected to be still on migration or to be summering non-breeder)
1 Species observed in breeding season in possible nesting habitat
2 Singing male(s) present (or breeding calls heard) in breeding season
3 Pair observed in suitable nesting habitat in breeding season
4 Permanent territory presumed through registration of territorial behaviour (song, etc.) on at least two different days a week or more apart at same place
5 Courtship and display
6 Visiting probable nest-site
7 Agitated behaviour or anxiety calls from adults
8 Brood patch on adult examined in the hand
9 Nest-building or excavating of nest-hole
10 Distraction-display or injury-feigning
11 Used nest or eggshells found (occupied or laid within period of survey)
12 Recently fledged young (nidicolous species) or downy young (nidifugous species)
13 Adults entering or leaving nest-site in circumstances indicating occupied nest (including high nests or nest holes, the contents of which cannot be seen) or adult seen incubating
14 Adult carrying a faecal sac or food for young
15 Nests containing eggs
16 Nests with young seen or heard

Partner sources

Check you partner source codes in your internal area.