Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optionally Have Segment Exporters Derive From AbstractSegmentExporter. #381

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
package org.vitrivr.cineast.core.features.abstracts;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Base64;
import java.util.HashMap;
import java.util.function.Supplier;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.vitrivr.cineast.core.data.segments.SegmentContainer;
import org.vitrivr.cineast.core.db.PersistencyWriterSupplier;
import org.vitrivr.cineast.core.db.setup.EntityCreator;
import org.vitrivr.cineast.core.features.extractor.Extractor;
import org.vitrivr.cineast.core.util.LogHelper;

abstract public class AbstractSegmentExporter implements Extractor {

protected static final Logger LOGGER = LogManager.getLogger();

private static final String PROPERTY_NAME_DESTINATION = "destination";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer the property name to be public.


/**
* Destination path for the audio-segment.
*/
private final Path destination;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this private object property documented and the following protected properties aren't?
I heavily suggest to document important information particularly on things that are exposed.
Our software stack has certainly room for improvement in terms of internal and external documentation and going forward, it seems to me it would be a good idea to aim for better documentation...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the path should be a folder? The documentation could be more precise.


protected String fileExtension;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would refactor this property to a protected abstract getter to force implementing classes to provide sensible values here


protected String dataUrlPrefix;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would refactor this property to a protected abstract getter to force implementing classes to provide sensible values here


/**
* Default constructor
*/
public AbstractSegmentExporter() {
this(new HashMap<>());
}

/**
* Default constructor. The AudioSegmentExport can be configured via named properties in the provided HashMap.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy-paster error: This isn't the AudioSegmentExporter anymore

* <p>
* Supported parameters:
*
* <ol>
* <li>destination: Path where files should be stored.</li>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would highlight in the documentation that its literally destination

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please clarify this?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have destination as the property key. I would expect that this is marked as such (e.g. with @code in line 57)

* </ol>
*
* @param properties HashMap containing named properties
*/
public AbstractSegmentExporter(HashMap<String, String> properties) {
this.destination = Path.of(properties.getOrDefault(PROPERTY_NAME_DESTINATION, "."));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not convinced that . is a sensible default here.
My suggestion would be $(pwd)/export to not fill the current working directory with individual files?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this and am fixing this. Just for the record, originally AudioSegmentExporter did use '.'

}

abstract public void exportToStream(SegmentContainer sc, OutputStream stream);
sauterl marked this conversation as resolved.
Show resolved Hide resolved

abstract public boolean isExportable(SegmentContainer sc);
sauterl marked this conversation as resolved.
Show resolved Hide resolved

@Override
public void processSegment(SegmentContainer shot) {
if (!isExportable(shot)) {
return;
}
try {
/* Prepare folder and OutputStream. */
Path folder = this.destination.resolve(shot.getSuperId());
if (!folder.toFile().exists()) {
folder.toFile().mkdirs();
}
Path path = folder.resolve(shot.getId() + this.fileExtension);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No guarantee that this.fileExtension is set. Also there is no sanitation aboutfileExtension should include the delimiter (.) or not.

OutputStream os = Files.newOutputStream(path);

/* Write audio data to OutputStream. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is data-type agnostic at this point, no?

exportToStream(shot, os);

/* Close OutputStream. */
os.close();
} catch (Exception e) {
LOGGER.error("Could not data to file for segment {} due to {}.", shot.getId(), LogHelper.getStackTrace(e));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested error message: Could not write data to file for segment {} due to {}.

}
}

public String exportToDataUrl(SegmentContainer shot) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation missing. Please add documentation

// create a ByteArrayOutputStream
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fairly sure that these one-line comments are not necessary and do not help understand the code, please remove them

ByteArrayOutputStream baos = new ByteArrayOutputStream();

// call the exportToStream method with the ByteArrayOutputStream
exportToStream(shot, baos);
sauterl marked this conversation as resolved.
Show resolved Hide resolved

// convert the ByteArrayOutputStream's data to a byte array
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fairly sure that these one-line comments are not necessary and do not help understand the code, please remove them

byte[] bytes = baos.toByteArray();

// encode the byte array to a Base64 string
String base64 = Base64.getEncoder().encodeToString(bytes);

// concatenate the data URL prefix and the Base64 string
return this.dataUrlPrefix + base64;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no guarantee at this point that this.dataUrlPrefix has been set

}

public byte[] exportToBinary(SegmentContainer shot) {
sauterl marked this conversation as resolved.
Show resolved Hide resolved
// create a ByteArrayOutputStream
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fairly sure that these one-line comments are not necessary and do not help understand the code, please remove them

ByteArrayOutputStream baos = new ByteArrayOutputStream();

// call the exportToStream method with the ByteArrayOutputStream
exportToStream(shot, baos);

// convert the ByteArrayOutputStream's data to a byte array
return baos.toByteArray();
}

@Override
public void init(PersistencyWriterSupplier phandlerSupply) { /* Nothing to init. */ }

@Override
public void finish() { /* Nothing to finish. */}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about closing the stream?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To put emphasis on this comment. I'd expect the finish method be abstract since the implementing exporter should flush and close its stream, not?


@Override
public void initalizePersistentLayer(Supplier<EntityCreator> supply) { /* Nothing to init. */ }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't in your case mean persistent layer database + outgoing stream? In this case I'd expect this to beabstract such that the implementing exporter is able to performe proper initialisation.


@Override
public void dropPersistentLayer(Supplier<EntityCreator> supply) { /* Nothing to drop. */}

}
Original file line number Diff line number Diff line change
@@ -1,41 +1,20 @@
package org.vitrivr.cineast.core.features.exporter;

import static java.nio.file.StandardOpenOption.CREATE;
import static java.nio.file.StandardOpenOption.TRUNCATE_EXISTING;

import java.io.IOException;
import java.io.OutputStream;
import java.nio.BufferOverflowException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.function.Supplier;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.vitrivr.cineast.core.data.frames.AudioFrame;
import org.vitrivr.cineast.core.data.segments.SegmentContainer;
import org.vitrivr.cineast.core.db.PersistencyWriterSupplier;
import org.vitrivr.cineast.core.db.setup.EntityCreator;
import org.vitrivr.cineast.core.features.extractor.Extractor;
import org.vitrivr.cineast.core.features.abstracts.AbstractSegmentExporter;
import org.vitrivr.cineast.core.util.LogHelper;

/**
* Exports the audio in a given segment as mono WAV file.
*/
public class AudioSegmentExporter implements Extractor {


private static final Logger LOGGER = LogManager.getLogger();

private static final String PROPERTY_NAME_DESTINATION = "destination";

/**
* Destination path for the audio-segment.
*/
private Path destination;
public class AudioSegmentExporter extends AbstractSegmentExporter {

/**
* Default constructor
Expand All @@ -56,7 +35,9 @@ public AudioSegmentExporter() {
* @param properties HashMap containing named properties
*/
public AudioSegmentExporter(HashMap<String, String> properties) {
this.destination = Paths.get(properties.getOrDefault(PROPERTY_NAME_DESTINATION, "."));
super(properties);
this.fileExtension = ".wav";
this.dataUrlPrefix = "data:audio/wav;base64,";
}

/**
Expand All @@ -65,12 +46,8 @@ public AudioSegmentExporter(HashMap<String, String> properties) {
* @param shot SegmentContainer to process.
*/
@Override
public void processSegment(SegmentContainer shot) {
public void exportToStream(SegmentContainer shot, OutputStream stream) {
try {
/* Prepare folder and OutputStream. */
Path directory = this.destination.resolve(shot.getSuperId());
Files.createDirectories(directory);
OutputStream stream = Files.newOutputStream(directory.resolve(shot.getId() + ".wav"), CREATE, TRUNCATE_EXISTING);

/* Extract mean samples and perpare byte buffer. */
short[] data = shot.getMeanSamplesAsShort();
Expand All @@ -85,12 +62,16 @@ public void processSegment(SegmentContainer shot) {
}

stream.write(buffer.array());
stream.close();
} catch (IOException | BufferOverflowException e) {
LOGGER.fatal("Could not export audio segment {} due to a serious IO error ({}).", shot.getId(), LogHelper.getStackTrace(e));
}
}

@Override
public boolean isExportable(SegmentContainer sc) {
return sc.getNumberOfSamples() > 0;
}

/**
* Writes the WAV header to the ByteBuffer (1 channel).
*
Expand Down Expand Up @@ -123,16 +104,4 @@ private void writeWaveHeader(ByteBuffer buffer, float samplingrate, short channe
buffer.putInt(subChunk2Length); /* Length of the data chunk. */
}


@Override
public void init(PersistencyWriterSupplier phandlerSupply) { /* Nothing to init. */ }

@Override
public void finish() { /* Nothing to finish. */}

@Override
public void initalizePersistentLayer(Supplier<EntityCreator> supply) { /* Nothing to init. */ }

@Override
public void dropPersistentLayer(Supplier<EntityCreator> supply) { /* Nothing to drop. */}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,14 @@

import java.awt.image.BufferedImage;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.List;
import java.util.function.Supplier;
import javax.imageio.ImageIO;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.vitrivr.cineast.core.data.segments.SegmentContainer;
import org.vitrivr.cineast.core.db.PersistencyWriterSupplier;
import org.vitrivr.cineast.core.db.setup.EntityCreator;
import org.vitrivr.cineast.core.features.extractor.Extractor;
import org.vitrivr.cineast.core.features.abstracts.AbstractSegmentExporter;
import org.vitrivr.cineast.core.util.LogHelper;
import org.vitrivr.cineast.core.util.dsp.fft.STFT;
import org.vitrivr.cineast.core.util.dsp.fft.Spectrum;
Expand All @@ -24,23 +19,18 @@
/**
* Visualizes and exporst the power spectogram (time vs. frequency vs. power) of the provided AudioSegment.
*/
public class AudioSpectogramExporter implements Extractor {
public class AudioSpectogramExporter extends AbstractSegmentExporter {


private static final Logger LOGGER = LogManager.getLogger();

/**
* Property names that can be used in the configuration hash map.
*/
private static final String PROPERTY_NAME_DESTINATION = "destination";
private static final String PROPERTY_NAME_WIDTH = "width";
private static final String PROPERTY_NAME_HEIGHT = "height";
private static final String PROPERTY_NAME_FORMAT = "format";

/**
* Destination path; can be set in the AudioWaveformExporter properties.
*/
private final Path destination;

/**
* Width of the resulting image in pixels.
Expand All @@ -53,7 +43,7 @@ public class AudioSpectogramExporter implements Extractor {
private final int height;

/**
* Output format for thumbnails. Defaults to PNG.
* Output format for thumbnails. Defaults to JPG.
*/
private final String format;

Expand All @@ -78,30 +68,25 @@ public AudioSpectogramExporter() {
* @param properties HashMap containing named properties
*/
public AudioSpectogramExporter(HashMap<String, String> properties) {
this.destination = Paths.get(properties.getOrDefault(PROPERTY_NAME_DESTINATION, "."));
super(properties);
this.width = Integer.parseInt(properties.getOrDefault(PROPERTY_NAME_WIDTH, "800"));
this.height = Integer.parseInt(properties.getOrDefault(PROPERTY_NAME_HEIGHT, "600"));
this.format = properties.getOrDefault(PROPERTY_NAME_FORMAT, "JPG");
this.fileExtension = "." + format.toLowerCase();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, format has to be a valid extension?

this.dataUrlPrefix = "data:image/" + format.toLowerCase() + ";base64,";
}

@Override
public void processSegment(SegmentContainer shot) {
/* If shot has no samples, this step is skipped. */
if (shot.getNumberOfSamples() == 0) {
return;
}

public void exportToStream(SegmentContainer shot, OutputStream stream) {
/* Prepare STFT and Spectrum for the segment. */
final Path directory = this.destination.resolve(shot.getSuperId());
final STFT stft = shot.getSTFT(2048, 512, new HanningWindow());
final List<Spectrum> spectrums = stft.getPowerSpectrum();

/* Visualize Spectrum and write it to disc. */
try {
BufferedImage image = AudioSignalVisualizer.visualizeSpectogram(spectrums, this.width, this.height);
if (image != null) {
Files.createDirectories(directory);
ImageIO.write(image, format, directory.resolve(shot.getId() + "." + format.toLowerCase()).toFile());
ImageIO.write(image, format, stream);
} else {
LOGGER.warn("Spectrum could not be visualized!");
}
Expand All @@ -111,14 +96,8 @@ public void processSegment(SegmentContainer shot) {
}

@Override
public void init(PersistencyWriterSupplier phandlerSupplier) { /* Noting to init. */}

@Override
public void finish() { /* Nothing to finish. */}

@Override
public void initalizePersistentLayer(Supplier<EntityCreator> supply) {/* Nothing to initialize. */}
public boolean isExportable(SegmentContainer sc) {
return sc.getNumberOfSamples() > 0;
}

@Override
public void dropPersistentLayer(Supplier<EntityCreator> supply) {/* Nothing to drop. */}
}