Change FileSourceAdapter to ResourceSourceAdapter [INT-188]

[spring-operator]

Dave Syer opened INT-188 and commented

Change FileSourceAdapter to ResourceSourceAdapter. It would be more flexible and easier to test if it was based on a ResourcePatternResolver instead of a directory polling. Actually maybe the name is still fine (people will be looking for a file source, so the name will help), but the pattern resolver trick would be nice to have.

---

Affects: 1.0 M3

1 votes, 3 watchers

[spring-operator]

Mark Fisher commented

Reopened since this issue is more extensive (and hence not a duplicate of) #4269.

[spring-operator]

Dave Syer commented

Agreed that it is not a duplicate, so thanks for re-opening. This one would be nicer for users (IMO).

Just a thought on the class name: we use File* in class names in Spring Batch because that's what people usually use them for, but the implementation is in terms of Resource (kind of an internal detail, once you know that you are just using a literal to set the basic property).

ResourcePatternResolver might be a waste of time as well because the ResourceArrayEditor in Spring does a great job of converting a resource pattern into an array of resources.

[spring-operator]

Marius Bogoevici commented

Dave, you're right - the current feature is a subset of this so they would have to be merged, eventually.

The way the file adapter works is that whenever a new file matching the criteria is created, it would be transformed into a message.

I think that ResArrEditor is applied too early in the process to allow for dynamically accommodating the creation of new Resources, so one would have to re-apply the Resolver with each polling operation.

[spring-operator]

Iwein Fuld commented

I had a quick look at the ResourceArrayEditor, but it seems to me that it is meant to resolve an immutable array of Resources? Would it be worth the effort to create our own implementations for constantly changing lists of files? It also seems to me that the current implementations for Resource** are very much aimed at scanning the classpath/filesystem once.

For inbound files (see PollableFileSource) you would not likely use classpath: and you would have to scan each poll. I've used a Resource for the input directory now so that users can use familiar patterns while configuring the inbound-channel-adapter, but I don't think using Resource** instead of File** for the internal listings brings much benefit.

Please correct me if I'm wrong.

[spring-operator]

Mark Fisher commented

I've removed a target version for this, since there seems to be some question of whether it makes sense to move in this direction in the first place. I'm not closing the issue yet, however, since it seems that the conversation may not be over. It should also be noted that the inbound File adapter has changed significantly since this issue was first raised.

[spring-operator]

Iwein Fuld commented

The main problem roots for this issue are resolved with refactorings in the FileReadingMessageSource (for 1.0.3). I've tried and tested it in my current project and the testing issues that we had earlier are gone. I've also experimented with reading actual files from inside the jar, and found that it's not a very practical idea to say the least. I have yet to hear a use case to support keeping this issue open for that purpose.

We can now read files from a directory on the test classpath without any problems, and for the input directory a classpath: prefix will also work (assuming the location on the classpath is not inside a jar).

I think it's time to put this issue to bed.

[spring-operator]

Iwein Fuld commented

I'm regretting the decision to go with Resource earlier. I've refactored FileReadingMessageSource and FileReadingMessageSourceFactoryBean to use plain File. The reason for this is that the directory needs to be a File for this to work anyway. Any niceties regarding the parsing of "classpath:" and "file:" do not belong in this class in the first place.

I'll extend the testcases a bit to make sure different types of configurations (especially around testing) just work and then I'll commit the plain old file version.

[spring-operator]

Iwein Fuld commented

So far it seems the only side effect of my latest refactoring is that the file: prefix is no longer needed in most cases. Loading from the classpath: still works. I'm going to commit this and close the issue.

[spring-operator]

Marius Bogoevici commented

Iwein has spent most of the time working on this issue lately anyway.

[spring-operator]

Iwein Fuld commented

Dave, I think we shouldn't fix this (any further). If you disagree, can you provide some examples that don't work with the current implementation and reopen?