source: josm/trunk/src/org/openstreetmap/josm/data/protobuf/ProtobufPacked.java@ 18431

Last change on this file since 18431 was 18431, checked in by taylor.smock, 4 years ago

VectorTile/MapBox/Feature: Significantly reduce allocations

The area tested can be downloaded with
--download=39.0637818,-108.5670233,39.0660809,-108.5620022.
VectorTile downloads was tested via the Mapillary plugin.

Differences (bytes, from IntelliJ Profiler in IDEA 2022.1):

Class Method Original New Difference
CommandInteger init 81,156,792 5,045,360 -76,111,432 (-93.7%)
Feature parseTagValue 354,995,472 143,200,680 -211,794,792 (-59.6%)
ProtobufPacked init 458,471,192 189,104,744 -269,366,448 (-58.7%)
ProtobufRecord init 492,731,416 142,510,112 -350,221,304 (-71.0%)
Feature init 1,437,014,520 550,452,376 -886,562,144 (-61.7%)

Most of the allocations came from the following sources:

  • Array copies
    • List#add (mostly fixed through the use of a ByteArrayOutputStream, as the lists were List<Byte>)
    • Enum#values (this is where the changes to Command and WireType come from)
  • Streams (fixed by using a traditional for loop)
  • NumberFormat#getNumberInstance (fixed by creating a static instance)
File size: 2.3 KB
Line 
1// License: GPL. For details, see LICENSE file.
2package org.openstreetmap.josm.data.protobuf;
3
4import java.io.ByteArrayOutputStream;
5import java.util.ArrayList;
6import java.util.List;
7
8/**
9 * Parse packed values (only numerical values)
10 *
11 * @author Taylor Smock
12 * @since 17862
13 */
14public class ProtobufPacked {
15 private final byte[] bytes;
16 private final Number[] numbers;
17 private int location;
18
19 /**
20 * Create a new ProtobufPacked object
21 *
22 * @param bytes The packed bytes
23 */
24 public ProtobufPacked(byte[] bytes) {
25 this.location = 0;
26 this.bytes = bytes;
27 List<Number> numbersT = new ArrayList<>();
28 // By reusing a ByteArrayOutputStream, we can reduce allocations in nextVarInt from 230 MB to 74 MB.
29 ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(4);
30 while (this.location < bytes.length) {
31 numbersT.add(ProtobufParser.convertByteArray(this.nextVarInt(byteArrayOutputStream), ProtobufParser.VAR_INT_BYTE_SIZE));
32 byteArrayOutputStream.reset();
33 }
34
35 this.numbers = new Number[numbersT.size()];
36 for (int i = 0; i < numbersT.size(); i++) {
37 this.numbers[i] = numbersT.get(i);
38 }
39 }
40
41 /**
42 * Get the parsed number array
43 *
44 * @return The number array
45 */
46 public Number[] getArray() {
47 return this.numbers;
48 }
49
50 private byte[] nextVarInt(final ByteArrayOutputStream byteArrayOutputStream) {
51 // In a real world test, the largest List<Byte> seen had 3 elements. Use 4 to avoid most new array allocations.
52 // Memory allocations went from 368 MB to 280 MB by using an initial array allocation. When using a
53 // ByteArrayOutputStream, it went down to 230 MB.
54 while ((this.bytes[this.location] & ProtobufParser.MOST_SIGNIFICANT_BYTE)
55 == ProtobufParser.MOST_SIGNIFICANT_BYTE) {
56 // Get rid of the leading bit (shift left 1, then shift right 1 unsigned)
57 byteArrayOutputStream.write(this.bytes[this.location++] ^ ProtobufParser.MOST_SIGNIFICANT_BYTE);
58 }
59 // The last byte doesn't drop the most significant bit
60 byteArrayOutputStream.write(this.bytes[this.location++]);
61 return byteArrayOutputStream.toByteArray();
62 }
63}
Note: See TracBrowser for help on using the repository browser.